#citizencosmos

  • Episode link: https://www.citizencosmos.space/notional

  • Episode name: Jacob Gadikian, sand, relaying & the interchain

Transcript of the Episode:

[00:01:41] Citizen Cosmos: Hey, everyone, welcome to a new episode of the Citizen Cosmos podcast! We have Jacob Gadikian, founder of Notional, a disciplined software development validator.

[00:02:07] Jacob Gadikian: Hi! It’s good to be here. So Notional was a uniswap airdrop and a computer. Our team now is 20 people. It's pretty incredible, actually.

[00:02:13] Citizen Cosmos: 20? Wow! I have to be honest before the listeners, I prepared zero questions for Jacob because he’s a great guy, and I thought the conversation is going to be free flow so this is why the conversation might seem a little bit lost it's not. We are on track. Please carry on with the introduction Jacob.

[00:02:53] Jacob Gadikian: So basically, in February of 2021, several things happened. One, I left Tendermint and began to research dexes for a very real-world physical commodities trading product, and since that's still in play, I'm gonna go into detail on that; that’s something I'm still consulting on. Initially, I looked at applying the tools and techniques in the GravityDex. Somebody along the way said: “Jacob you really need to talk to Sunny, he's building something, and he had invited me to a conference in 2019, but my son had just been born, so I wasn’t able to attend. It was the last time I had spoken to him except for once at Tendermint. I sent him a Twitter DM, and he told me we need to talk. So he and I had about an hour-and-a-half-long call where he introduced Osmosis to me, and honestly, my comprehension was very low. I never got into Ethereum DeFi except, of course, enough to deploy something very amateurish onto Ethereum, which was enough to get me a very significant uniswap airdrop. I had spoken to Sunny, and then one day, I saw that Osmosis was about to launch, and I threw in a Genesis transaction. Our uptime was initially terrible, I was validating from my family's old apartment, and we were getting 80% uptime. Initially, I created just some new documentation around seeds and peers and work down our network a bit and got that up, and made it very stable. That was when I first built an open MPTCP router and linked that up to another IASP in Vietnam. After that, I explored relaying. I had my Osmosis validator, and we started to get requests to back up on relaying. So I did the majority of relaying that summer and this actually got me down. It's now a 14-month-long rabbit hole in Cosmos performance, so my computer has a ridiculously fast hard drive; it's an Intel p5800x. Anyhow, U quickly learned from observing that rate actually slowed things down, and because of the stripe size on it, the chain would keep trying to write all these small files smaller than the 32 kbs but the striped size of the rate was 32 kbs. I was relaying against our validator node, you should not do this if you want to learn how to relay. Please do not follow these directions it was not safe. I had the relayer running I had three or four validators on that computer. I noticed that when IBC packets would build up, the validators would miss blocks, and I was relaying a fairly significant portion of the whole IBC network packets. I was also trying to figure out why our packets would come in faster than Cephalopods packets, they were trying to figure that out too. Also, that was not consistent, but there was just this tendency, turns out that Cosmos chains would just show up disusage and it's only in the past month that we put this issue to bed. There are still a lot more performance things to fix in Cosmos, but at least on the database layer, with three Notional team members, we refactored heavily the database that the Cosmos SDK uses tore almost everything out; it’s 75% lighter, and introduced a new data bates called Pebble DB. Now, sometime around September 2021, I stopped doing things dangerously and began to formalize operations. I separated validation and relaying. Validation was strictly on-site, and relaying was strictly at Hetzer. I think this is about the time that we really began to grow. In May of 2021, I met an amazing young individual, and he built Notional’s engineering team. They are technology students, and so they began to help with the process of formalizing operations and making things safer. To do that, we had to write code. The infrastructure operations weren't enough, we could get the fastest disk in the world and not get much of a speed boost from it. We knew that we were really dependent on speed, particularly for relaying and then on faster chains. Because that thing is so fast, there's an additional operational burden there. Evmos is fairly similar, it's just a very fast blockchain, and we began to specialize in that. Then, along the way, in about January 2020, I had been working on Juno I was told that there’s a chain they're going to build with StarPort, and it could be launched on the StarPort network. That part didn't play out, but what I would do for 10 months, I'd go between Juno and the StarPort template, and I'd update the template every time that I saw that Juno had sort of fallen behind the latest versions of the Cosmos core libraries. So by the fall of 2021, we were doing more and more with Osmosis, and we were collaborating with some teams although I'm going to give credit where credit is due, even though it's as controversial, we never really got paid by Terra but a lot of the database stuff that we worked on definitely came from a long string of interactions that included Do Kwon because we had just launched Chihuahua chain, I was working on the database yet again. I thought it would be a good idea to record a video barking like a Chihuahua about the speed of the chain, and I did and shared it.

[00:11:47] Citizen Cosmos: Just to point out, nobody got paid by Do Kwon; everybody got misused by him. Even if you got paid by Do Kwon, you probably put it back in a way.

[00:12:04] Jacob Gadikian: Yeah, there was a certain recycling effect. Luckily, we weren’t directly exposed. Of course, the indirect exposure was still horrific. A lot of Cosmos went down by 80-90%, and during that week, I was watching the chain. I still suspect that the chain was attacked at the same time the chain's economy was attacked. Really weird things happened across their validator set.

[00:12:38] Citizen Cosmos: I must say that it’s great to hear your side of things after different conversations with guys from Cephalopods.

[00:13:04] Jacob Gadikian: so look, I'll take a second here to praise Cephalopod and what informal did for IBC in the beginning. IBC is very huge and complex and probably needs to be. The key base level of IBC is 2 tendermint chains exchanging light clients. After that, the protocol pretty much mirrors TCP/IP and it’s a good analogy for anybody who is technically looking into it. At the time, it was Hermes that worked well. Since then, Strangelove has done a very deep refactor on the go relayer. Go relayer version 2.0 was very reliable and very performant. In the summer of 2021 though, it was just Hermes and a lot of things broke. Things broke constantly. You would see channels get backed up. Nobody knew why; we just had our guesses.
I think the thing that I was right about was the influence of hardware. One of the things that we figured out or added evidence to was that there were situations where Hermes would work well, but I don't think that we know to this day what caused some of those issues because we decoupled IBC from the SDK in version 43. Version 43 was pulled back, and then 44 was released. Osmosis ran 44 for a bit. I helped to upgrade Osmosis to 45, and I think that's when IBC 2 was released. I mean that in the original Osmosis, which kicked off the big explosion in IBC, many issues, I believe, are still undiagnosed. We had a bunch of guesses I’m sure that those guesses fed into both the IBC spec and then the IBC code, and now I would say that the big deal presently is that we’re integrating CosmWasm and IBC to a much greater degree, so that if a chain is a computer and another chain is a computer, then you have applications on each computer. Those contracts can talk to each other, and that's the stage where we are now. So how did we get to 20 people? Along the way, our focus shifted. Initially, we were mainly an operator. We learned how to run relayers and run them well, we also helped to teach people how to run relayers. We then began writing code that led to different engineering contracts throughout the ecosystem. Presently we support Osmosis, Juno, Pylon’s craft economy. We’re pushing out SDK-46 across the ecosystem and working on performance, so that's how we ended up at 20 people, and we will likely move to a new office this or next month, and that's the story.

[00:17:56] Citizen Cosmos: There is one thing here I would like to ask you. I would love to talk about what makes you tick and keeps you motivated in governance, in trying to make things fair for everybody, etc. What is the push forward?

[00:19:40] Jacob Gadikian: Frankly, some of the things I've seen around Cosmos self-development are not up to the Osmosis standard, but I hate for anybody to think I'm just baselessly protecting Osmosis. The Hub has kicked off Cosmos, and many people probably don't know this, but I was a validator, and I noticed that the Hub had a much smaller-sized environment. If we’re talking about delegations, this is getting taken care of. There was a validator doing some nasty things to Gaia and ICF, and they withdrew their delegation, which was the right thing to do. So I noticed that the allegations were off and that builders were not being supported like they were on Osmosis. Furthermore, if we go to something I've spoken about many times on Twitter, there's just one random string, no name, no contact, 20% commission, no governance participation, and frequent downtime. Everything is bad about a validator. But it still had a very effective ICF delegation, and I questioned that, unfortunately, along the way of questioning, which seems to have harmed relationships. I hope that that can be amended. I'm going to defend myself and say that those things need to be taken care of, and they were not addressed in a serious manner when I first approached them. I first approached them in a minimal semi-private context, and I was told to let the professionals handle it - they didn't.

[00:21:28] Citizen Cosmos: I love that sentence: let the professional handle it.

[00:21:39] Jacob Gadikian: Yeah. Well, that's more or less how I felt about it, and I think that in a healthier environment, things like that would not harm relationships. I found and eventually resolved only through working with the code reviewers; one of them accused me of code of conduct violations for tweeting too much during proposal 69. I don't want to name names and am not interested in further damaging relationships. I'm interested in building relationships where we can say that there was a pretty serious problem with the Hub where it would always pass. I want to give credit where it's due, so the Cosmos SDK team, and IBC team, really did an incredible job helping me get code that touched on the usability of SDK-46 because Notional invested a tremendous amount of energy and money time in building out chains on SDK-46 and one of the byproducts of that so we've put up the only chain on 46 that's Pylon’s and that will soon be going from testnet to mainnet, it's a chain that allows cards to be accepted. We have been building a craft economy on SDK-46 since the fall of last year, and more recently, there was some highly inaccurate information spread about governance. I want to have fact-driven conversations about all this stuff. I feel that politics have been taken over, and I'm not trying to hurt anyone's feelings. I genuinely care about the Cosmos Hub and Cosmos overall, and I think that because I raised these issues, Notional has been treated wrong and at a very disappointing scale. We rewrote tmdb, but there has been no recognition of that. We have created tooling to automatically find unclosed iterators in chains which turned out to be a relatively common issue that was affecting database performance all across Cosmos, and at the end of the day, if we're kind of cast aside, will just go and work out on Osmosis and Juno. In my opinion, this has just been a misuse of our time, and I do not feel that it is disrespectful to point out that during all the debates around CosmWasm. When you enable the CosmWasm chain, you double its system resource consumption. This is just that's just how it is. We love Cosmos at Notional. In terms of features and functions against CosmWasm on the Hub, my views don't match Jay's views there, and one of the things that were repeatedly tossed out during the debate was that I was some kind of a proxy for him. He's the guy who taught me all of this. I've had many teachers, and that's how you learn. In Cosmos, there is a very non-formal mentorship. It was Jay, it was Alessio. Other members of the Tendermint team, including Dennis - head of StarFort, was Jack. Jack noticed I was doing a lot of IBC traffic. He called me one day out of the blue and said: “How are you doing this? I don't think you know how it works.” Then he spent about two weeks making sure I knew how IBC worked. That's a huge deal; he invested a lot of time. After that, it was Dave from Osmosis. When I began to work on the chain, he spent tons of time making sure that I knew what the hell was going on under the hood because I came in fairly green. I’m a former history teacher and Bitcoin miner. I've learned a lot of technical stuff.

[00:26:30] Citizen Cosmos: This is interesting. I want to know your story, man. I'm putting you a little bit off the track, but I would love to hear how you began loving hardware and where the love was born. So what’s the story? Where did it start?

[00:27:21] Jacob Gadikian: It's a little longer, but I'll go right into it. I was teaching history in China. It was a private high school program where the students would eventually go to school in the United States, and it was cool, I got to do the curriculum design. But during that time, Xi Jinping became president of China, and stuff changed quite a bit. I don't know much about Chinese politics, but two notable things changed. If you work in a school in China, you have contact with the government by necessity. This school was considered a pilot program, and there were just mixed opinions. Secondly, the curriculum teaching style was affected, and the politics courses became mandatory from my perspective, I wanted to learn about all that stuff, but that's not my point of view. I found it very interesting because one thing that did happen in China was rapid human advancement at scale. That's notable and probably shouldn't be ignored. Any time you're looking at what China is today, you should be thinking about mega cities, you should think about the advancement in human development, and you should probably also be thinking about the double-edged sword. The negative part about this for me is that for a time, I got separated from my family by the great firewall. This is probably how you and I met online, by the way. So it's also how I started to use Bitcoin. China made the great firewall far more strict, and my interest in hardware and electronics, I’ve been doing that all my life. So in China, I was testing and collecting all kinds of weird, interesting Chinese electronics because it's all made there, so you can get some neat stuff there I met with several companies based in Shenzhen whose products I like. So in the middle of my first year of teaching, Xi Jinping took power. For a brief time, I could not easily contact my family; there were ways to do it, of course, but it became difficult, and we certainly couldn't have a call like we’re having now. So I learned to build VPNs because the commercial one was not reliable. I wasn't concerned with privacy either, I was more about connectivity. This is what led me to Bitcoin. I learned about MadeSafe, I participated in the ICO, and somehow it led to bit shares of Fuzzy. I came on the Fuzzy show in 2013 or 2014. By the end of my second year of teaching, I had invested in Bitcoin, rolled in all the coins, and lost a great deal of that play money, of course. Then, I decided to quit teaching, moved to Shenzhen, and made open-source hardware because these wallets and private keys are the hardware we run. The keys are always vulnerable, and there we got about a six-month crash course in why hardware isn't open source or, more accurately, why open-source software hardware is crazy hard. If you look at the arm CPU market, people license the arm design those drivers are proprietary. Intel is relatively similar they protect things at different levels in the stack until drivers are all open, but Intel’s PCB reference designs are all closed, and then certainly, the semiconductor material in both cases is closed. All of that failed, and I entered weird political stuff again; my visa ran out during the umbrella revolution, so I went from Shenzhen to Hong Kong. I was down there for a week or two, you could taste the pepper spray in the air just about everywhere, and it looked like it was going to turn violent, and in many ways, it did turn violent, of course. But I felt it could easily get more violent than it already was. Additionally, there were counter-protesters. They were doing things like juggling knives in menacing ways on street corners. Eventually, I realized this wasn't going to stop. The government offices are closed, and I can't renew my Chinese visa. If I'd stayed another week, I probably could have had it, but I eventually decided this was not going to work because I was just there to renew my visa; it was going to be a very straightforward thing, and I flew down to Saigon where I failed again and ended up in Hanoi for two and a half years, and that's how I ended up coming here again, this time I got stuck here for Covid, and then in the middle part I visited Cambodia for Steemet South East Asia meet up. It was just a small meet-up that another Steemet user made but me, and my colleagues were writing on Steemer and making great money with it, decided to go to Cambodia. If you check my Steemet post, you'll find that I found some bugs, and what it turned out to be is Steemet using the JavaScript 256k1 implementation. I changed my view, I thought it was insecure, it's non-canonical. So there's a way that you're supposed to use that library, Steemet doesn't use it that way, and they're very prescriptive about how to use that library, and that's how I found Jae Kwon. I was researching proof of stake consensus mechanisms so that I could try to make a replacement for Steemet, it was called down, and I was having mental health trouble, but I didn't know it, and that's the scariest thing about going crazy. You will not know if you're going off the deep end. I was, but I didn't know it. I probably had undiagnosed bipolar for 15 years. Long story short, it all culminates in that crazy thing with me stealing a giant truck full of sand in Phnom Penh. I was delusional, I don't precisely how or why I felt that I needed to steal transportation, then drive that transportation to China. These were my overriding priorities in life, greater than food, greater than anything I attempted to do for about a week, and I was eventually successful. That's probably not a very good thing. I nearly drove into a parade, but luckily I had no murderous intent, and I stopped the truck. Then I got my ass kicked by a giant mob of people and went to jail. One of the most interesting and heartwarming parts of this story is that the truck’s owner came to see me in jail, took one look at my crazy ass, and within a day, the U.S. embassy picked me up and took me to a psychiatric hospital. There they tried everything except for actually what helped me. To this day, I religiously take lithium but there they put me on every downer type of drug that you can imagine, and it didn’t work, but then they got me into an airplane after about two weeks, and so I flew from Phnom Penh to Buffalo with a doctor who was feeding me pills the entire time, and so he got me to the Buffalo psychiatric center, and this stuff genuinely helped me, I wouldn't be here today if it if things hadn't gone in this way.

[00:40:06] Citizen Cosmos: I can relate very much to the story. The first time I was diagnosed with bipolar was 18 years ago when I was about 17-18. I think I decided when I was 18 that will be my prerogative and battle, and I refused to take the pills that I was prescribed, all the downers. I never did, and I think that helped me keep it. But again, with bipolar, it's a very loud sentence to say because you have very different levels of bipolar disorder and how it develops from the first to the fourth stage. I'm trying to connect here everything, your history teaching, the Chinese government changes, the love for the hardware, and what I see and hear there have been misjudgments which you saw with the Chinese government and how they started to treat the inability to contact you family and that to me calls as if the governance goes wrong, if nobody says anything in the right time and doesn't try to do something, that's what's going to happen.

[00:41:55] Jacob Gadikian: I should expand on that. So I was in an Iraq war protest. It was the dumbest, most futile thing I've ever done. Technically speaking, that's the first time I was arrested, but they didn't book it. I was in Chicago, put in plastic handcuffs, taken to the other side of the city, and jumped out, which was pointless. What I mean is what I was doing at the time, carrying around and talking about war, was pointless because there was a giant apparatus of lies backing up that conflict. I am sensitive to governance issues and the accuracy of the information that affects governance, and maybe just give you another dot to connect. Going back to the whole mental health situation, I made it miraculously and, in my mind, statistically unlikely recovery. Most of the people that I met in the mental hospital I don't think that they recovered, and they got the same treatment but did not recover. But I did. I Had a lot of help from my family, but here's a deal. I began to take lithium in the summer of 2017, and by the fall of 2017, I was figuring out how to raise venture capital when you have just stolen a giant truck full of sand. We were successful. Drone energy raised a million dollars from all kinds of investors, and I believe has raised another round now. The business worked, and that's one of the reasons why I'm so comfortable speaking openly about that experience because, for good reasons, investors will look into your background and will ask you questions about it. It's a substantial amount of money, they want to know that you're not going to invest in sand trucks instead of Bitcoin mines. So I learned to tell the facts on that. It's a very relationship-based business; you're seeking capital to purchase equipment and sourcing the business's energy contracts. I ended up doing a lot of that, I learned to be very direct about things even if they're uncomfortable, and it's not especially the first 75 times because when you raise venture capital, don't talk to one investor. You don't do this once you do that many times and find the ones you click with, so the first 75 times were hard, but after that, not really. It's just another piece of my history that just has to be incredibly fact-driven. Otherwise, why would anybody trust me? Frankly, if I were to try to hide that, they shouldn't trust me.

[00:45:38] Citizen Cosmos: I’d like to say that all you just said come as battling bipolar and succeeding because, to me, those swings that are uncontrollable are something I want to control. I don't trust those swings, and I don’t want to trust them. And that's why I would rather be direct and say this is what I am. These are my minuses, these are my pluses. This is how I’ve been living my life for over 20 years, and it immediately calls back the things I see in the world and how I interact with the planet, and I understand it very well.

[00:46:55] Jacob Gadikian: That's interesting that we both share that. I failed over and over until I started taking lithium. I still experience bipolar sometimes, sleep is very hard. I don't have the world's most regular or normal sleep schedule, but I sleep every day, and I wouldn't need to I'm not sure that most people can relate to that but I could just keep going, but they said that's dangerous. I look after the whole sleep thing. My wife checks in on it. That's important, I give her a lot of credit for my being alive. She chased me around Cambodia for about 6 months, getting me out all kinds of messed up, crazy ridiculous shit. So I got back up on my feet in the States and started to make plans to marry her. So we’re now married and live in Hanoi. We have a kid and trying to have another kid. So to answer your question, though, I have seen that since I was young. I was 18 in that Iraq war protest, which left a big impression on me.

[00:48:54] Citizen Cosmos: You're gonna laugh, but the first protest I went to was in 2003 against the war in Iraq. That’s very ironic.

[00:49:22] Jacob Gadikian: So now, just about being rigorously fact-driven, and then in the blockchain space, you only talk about this issue because it's solved. This was also true of the Cosmos SDK, with several CI jobs that hadn't been fired in years. It was a program to check the code, one of them is called a Linter. The result was that no matter what you did, various things that were supposed to check and test the code would always tell you everything was all good. We fixed the SDK and then moved on to Gaia, and I didn't want to be a pariah because I was insistent that the delegations were screwed up and were harming governance. It affected Quorum quite a bit because of the lack of governance participation among recipients of ICF delegations. If I found it I can't leave it to the professionals, I must be the professional because it wasn't being addressed. We need to focus on the code, we need to focus on transparency, and go public.

[00:54:12] Citizen Cosmos: I think it’s important to hear an unbiased opinion. But regardless of that, we haven't opened up a lot of things, but I would love to ask you about your motivation. What keeps you from wanting to get off bed? With all you mentioned, it’s not easy for anyone with a struggle like yours.

[00:59:27] Jacob Gadikian: I guess it goes back to the Iraq war protest. I do think that governments worldwide are fairly fundamentally flawed and that transparency would solve a lot. I recently voted for a 5% minimum commission on the Hub and am a big supporter of markets. One thing I've found frustrating throughout my career as an entrepreneur is the incorporation and capital formation processes. I think most start-ups fail and lose 100% of their investment capital. It's only a moral issue when there isn't a genuine effort; in that case, that's like fraud or a scam, and chains are not companies. There’s also Dig. Dig is a project based on readme files: we’re going to put physical plots on the chain, we're going to sell those physical plots of land, and we're going to excel because of transparency. That project suffered tremendously because of the conflict in Ukraine. We overinvested in that market, but there is a particular reason. Probably if I was faced with the same set of decisions again, I’d probably make the same decisions because they had the legal structures that they had put in place just before the war for martial law was put into place and made the Dig project o100% legal so we would have been able to do everything, the whole nine yards without question. So at present, we've upgraded Dig SDK-46, we have added CosmWasm to Dig, and we’ll be upgrading it in the next couple of weeks as soon as IBC is finalized.

[01:02:22] Citizen Cosmos: Back to the original question. What does keep you from getting out of bed?

[01:02:29] Jacob Gadikian: It's just the creation of transparent systems. I'm a true believer in the style of governance that we have in Cosmos chains even though it's messy and imperfect, I think the current setup can be extended a great deal. One of the reasons I love SDK-46 is because it did extend governance. We can do that more in 47.

[01:03:47] Citizen Cosmos: On that note, I would like to wrap this episode up because you had an important point that somehow concludes what we’ve been talking about. Thanks very much, and hopefully, we can speak again!

If you would like to support our mission in creating educational content and aligning the goals of different communities, please stake with us here)

Join our community, to build a future where communication is decentralized. May the code be with you!