Link to Twitter Space: https://twitter.com/i/spaces/1jMJgLQZOyPxL

Links:

Citizen Cosmos
Did it did. Can't. Hey, Andrew. Hey, man. Can you hear me? Okay, great. So we have sound. That's good. And lets approved Dan as well. Dan Can you hear me too? Say something. I'm good, Dan. Okay, great. So let's give it, like, some chit chat to the know, because it's going to take some time until people join. Let's hope that we have live ... some people.

Citizen Cosmos
And if not, we will, of course, promote the record in as much as we can. But still, regardless. So for before people start to join we start chit chat, maybe you guys want to tell me how you are. How's the weather?

Dan
Oh, yeah, sure. Doing good. I love Fridays. Really nice weather here in Maryland. We had a little bit of, like air quality issues because of the fires up in Canada the past few days. But it's getting better now.

Citizen Cosmos
I heard. I heard that you guys survive in breathing.

Dan
Yeah. I mean, Maraland's not very close to it, but other places like New York got hit a lot harder.

Citizen Cosmos
Oh, okay. Andrew, what about you? How is your. You Alive?

Andrew
Yeah, I'm out in Colorado, and, yeah, we were dealing with the fires a couple of weeks ago, and now it's started to shift east. But no. Yeah, past couple of weeks, it's been raining a ton, just on and off throughout the day, which has been great to green things up. So very beautiful out here in Colorado. A nice warm day, too.

Citizen Cosmos
That reminds me of that song. We didn't start the fire, but okay, let's not go into those jokes right away. Go. And that's great, man. I mean, we've been having some. What about you? Yeah, I've been. I was going to say, we've been having some strange weather for for for us, because I live on an island and in the ocean, and we've been usually like, it's quite hot for this time of the year.

Citizen Cosmos
But yeah, it's been raining a little bit, which is okay I guess for subtropics. Yeah, but oh by the way, for people who slow to join in before, it's just. Andrew, sorry about to interrupt you there. This is not a weather forecast. This is just us talking. So, Andrew, so back to you.

Andrew
Oh, no, no problem.

Citizen Cosmos
I thought you were going to add something. So, guys, for. For everybody. Who's Just starting to join in? Just to I'm going to say I'm going to repeat this probably several times before we kick off. And so this is going to be a validator focused space. And we have kindly the two Big Brains at Horcrux, Dan and Andrew.

Citizen Cosmos
And they will introduce themselves, of course, much better than that. But yeah, the idea is to kind of unravel the mystery, which is Horcrux seems is to a lot of validators. And to be honest, again, I will repeat this again, but but for the people who already join so it's not silence there I was having an episode with Crypto Crew a Validator team, and I realized when we were talking that it still, even to the guys who use it, can be like, you know, it still sounds intimidating sometimes.

Citizen Cosmos
So and I hope that Andrew and Dan are going to help us to understand that it's not. And if people should use it. So yeah, by the way, Andrew, Dan you guys are really want to like not introduce yourself yet, but maybe just in general Horcrux, chit chat, I don't know whatever you guys want to say and.

Andrew
Yes, I mean it's an exciting couple, couple of weeks for us gearing up towards this Horcrux V3 release which we released two weeks ago. We have a blog post that just went out yesterday. We're really excited to talk about, you know, some of the features that we introduced and simplifying things for validators even further.

Citizen Cosmos
Dan you want to add something to their you were starting to talk Yeah.

Dan
I think Andrew covered it but yeah we're excited to talk about Horcrux. Some of the changes that we've made, especially in terms of like making it easier and safer.

Citizen Cosmos
Awesome. Let's just play the couplet on Sorry, I'm going to be one of those annoying hosts and make everybody wait a couple of more minutes so hopefully there will be more people joining in. We have left the message and several validator groups and chats. Hopefully it was noticed. Some people did actually.

Citizen Cosmos
Actually, there was a lot of interest, so im surprise to be honest, to see yet so little people because a lot of people are really Horcrux finaly somebody is explaining somebody is doing it now let's talk about Horcrux. So I'm looking personally forward to it and I understand how it works. I have tested it before. Now, currently Citizen Cosmos, I'm going to be honest, we don't use it.

Citizen Cosmos
And but yeah, this is, I think, something to be fixed. And hopefully my understanding, personal understanding will also improve a lot today. I do have some of it, but of course I'm not Dan or Andrew and hopefully there will. I think it was it Dan was it you had the name, the original, the what was it, the Muggles and the Wizards or was it Andrew.

Citizen Cosmos
Who was it with a name.

Dan
Yeah, that was me. I like Harry Potter fan. So I came up with a little catch phrase for it. Nice. You want to you want to read that tagline? Are you ready for it? it's something like Horcrux shouldn't be a soul splitting decision. It's for all wizards and muggles.

Citizen Cosmos
I love it. This soul splitting thing. And it sounds like puns intended, you know? So.

Dan
Definitely.

Citizen Cosmos
Okay, guys, I think it's like 5 minutes pass. I know there isn't that many people and some are still joining in. Let's start with the intro slowly and then we will start. I mean, it's a beautiful Friday for me. I hope it's a beautiful Friday for other people, too. So even though we are like here developing space, you know, we still want to let you people off today yet not tomorrow.

Citizen Cosmos
So without due ado, guys, I'm going to ask Dan and Andrew to introduce yourself one by one. Andrew, do you want to kick in first and introduce yourself a little bit?

Andrew
Yeah, definitely. So I'm Andrew Gouin. I'm the director of engineering at Strange Love. Horcrux was actually one of the first projects that I started coding on when I started. But yeah, you know, I've spent about three years within the Cosmos ecosystem as a bystander from from the Stargate and Prior, and, you know, it's been great to get plugged in and get hands on with, with the core stack here.

Andrew
But yeah, I spent some time. I was, I mined ether for, for many years until that was no longer profitable and you know, did tooling here and there. But yeah great to great to be on it's great to be a part of a strong team that's contributing all around the ecosystem.

Citizen Cosmos
Yeah. Mean, just before Dan's going to start his intro, I just want to add. Yes, Strangelove. have been great, really. I don't think just for the ecosystem because the things you're doing can be used beyond Cosmos, in my opinion. And yeah, I've had the privilege of speaking to Jack many times and to Tyler as well. And yeah, really great thing, guys.

Citizen Cosmos
Well done. Sorry. Dan. Didn't want to steal your intro, please. My apologies.

Dan
Oh, yeah, sure. So I kind of do two things off in a day for my day job. I'm at a strange love on Andrew Steam, so I help run the infrastructure over there. So validators and Relayer is primarily. And then for my night job, I have my own company that I'm working on growing called Defiant labs. We build one product called Sycamore Tax that focused on cosmos taxes, and then we also are trying to do some validator stuff and grants.

Dan
Yeah, Andrew really taught me pretty much everything I knew about Horcrux. And over the past year I've got an opportunity to kind of get comfortable with it and understand it really well. We're actually going to be doing a workshop on Horcrux with Thor hax, so if any of you guys are interested in that. We'll hopefully get to share some more info on that soon too.

Citizen Cosmos
Oh, so I was going to say, when you said by night I was going to I was hoping for some Batman stuff there, but it was close enough. Close enough and I'll have it your own project that takes as much space as probably Batman takes the battle crime, especially if you're doing it at night times, guys. So I'm going to start slowly with, like, the questions and we're going to slowly go into it.

Citizen Cosmos
But just before one last thing, if anybody has a question, raise your hand. I will let you, of course, speak or if you're too shy and don't want to ask a question using your voice, you can write to the DMs of the Citizen Cosmos are open. So. Right. And maybe wink at me or something like this here. So I'll look at them, but I'll probably see them.

Citizen Cosmos
But just in case, I'll also that you can chat, right? Well, it's not really a chat, but you guys know how it works. So I'm not going to go into how Twitter space works anyways. If you have questions, please don't be too shy to ask them. And that is exactly the time. So anyways, Horcrux, so am I don't know how regardless whether again you're listening now or this is the recording for you.

Citizen Cosmos
I don't know how much you're familiar with Horcrux, but this is something that I personally discovered. I don't know, maybe last year, I believe or so, and I don't know exactly. And to me, the description that the most sensible descriptions are not sensible. I apologize for it. The most precise description, the easiest to understand is it's basically like sentry for the private key of the validator, same architecture kind of thing.

Citizen Cosmos
So which what it means is basically helps the validators to improve their security by well, I was going to say twice, but it's a lot more than that, to be honest. And it's not only for the validators to be honest, because a validator in my understanding who uses Horcrux should also appeal a lot more deligatores for for a number of reasons, including security, slashing and security risks and so on.

Citizen Cosmos
So this is my understanding now. But of course, Andrew, do you want to kick in and explain what the overview of Horcrux?

Andrew
Yeah, absolutely. So, you know, as a Tendermint validator, what is your main concern? It's that you have this private key and with the standard architecture it means you have to run that on a single node. You know, with the minimal architecture, that single node is synching blocks with the chain and it's signing on the same machine. You can take one step back from that and introduce sentry nodes, which means now you have nodes in front of that validator nodes that are synching blocks and sending the blocks sign requests to the single validator node.

Andrew
But you still have a single point of failure for your private key. And so the availability and keeping your key safe are really kind of your main concerns there as validator. And where Horcrux steps in is it allows you to take that private key, make multiple shards from it, and load it onto multiple cosigner machines. And with that you have the ability to configure a threshold of, you know, how many of those machines should be required to assemble a full signature for the block.

Andrew
So, you know, this introduces fault tolerance as well as just that that key security of requiring multiple pieces to you to assemble the full key. And so yeah it's that's really you know the benefit of Horcrux is you're SRE can sleep more peacefully at night knowing that you know, a machine can go down within the architecture and you're not going to be missing blocks.

Andrew
And in addition, you know, so with the key, the main concern there is that you don't want to double sign you don't want to sign the same payload for the same block twice with a different signature because that will tombstone your validator or, you know, essentially destroy your validator on chain. So validators, if that happens, that means that you have to create a new validator and try to win over the delegators to switch over to your new validator.

Andrew
So, you know, that's that's a worst case scenario. We definitely don't want that to happen. So Horcrux really takes the pain out of that, where now you have a whole separate cluster from your sentries where you can manager your key shards and you know your point, your your Horcrux clusters to the sentry nodes so that any of the maintenance operations you do on the century nodes doesn't affect your your remote signer cluster.

Andrew
So it lets you keep your your key material fully separate from the nodes that are synching blocks that are interacting with the chain. And you know, that lets you have that extra level of security. And so yeah, with that double sign protection, all of the nodes, all the cosigner nodes in the Horcrux cluster are keeping state of what is the highest block that that I have signed and all of those are keeping that stack individually.

Andrew
And then they also have a mechanism to talk to each other to come to consensus on what is the highest block that we have assigned as a entire cluster. So with those two two gates of double sign protection, you can be guaranteed that the Horcrux cluster, no matter which cosigner is being asked to sign the block, that it will not double sign.

Citizen Cosmos
Okay. Now you said that people can sleep safe at night. Delegators, of course, apart if it's Dan because Dan is building his own project at night. I'm joking Dan do you want to add anything to the intro about Horcrux that Andrew has said that was just perfect. Sure.

Dan
Well, maybe I'll just like more summarize what he said. So, yeah, the way that I look at Horcruxes, there's like three parts to it. So the first thing is that it is a remote signer, so it decouples your signing process from your block indexing process, which is really nice. And what that allows you to do is it allows you to, like, treat your nodes almost as throwaway-able because Horcrux itself will take care of keeping track of all the signing state.

Dan
So you never have to worry about backing up like your validator state dot JSON, none of that. You can just if there's a problem with your node, you can just delete the nodes, spin up a new one from Snapshot, tell it to connect to your Horcrux cluster and it's up and running and there's no risk at all. That's the first advantage is that it's a remote signer.

Dan
There's other remote signers out there to like TMKMS. The second advantage to Horcrux is that it adds high availability. So it allows you to split your signer into multiple locations so that you could put like one on the East Coast, one on the West Coast, one out in Europe or something. And if there's ever an outage in one of those locations, the other two are still able to create the key material needed to sign. thats second advantage.

Dan
And then the third advantage is the security aspect of it. So the key is never located in one location. It's spread through multiple machines. So if you wanted to get access to that key material, you would have to compromise multiple nodes in order to recreate that key. And that's the the third advantage.

Citizen Cosmos
And I never even thought about the third advantage until you said it's now. I'm going to ask questions, which are two you guys probably going to seem, I mean Dan and Andrew, like really obvious, but this is what I have gathered about when over the last six or seven months when I've started to mention Horcrux on my podcast, our are on on our Odyssey streams, and I know this is what I've been getting, so I'm going to be asking you those for some I think I understand more or less.

Citizen Cosmos
But again, this is not for me. Hopefully this is going to help people to demystify this. And so the first question, will this increase my... my cost of infrastructure and how much will this increase my cost of infrastructure by?

Dan
There really depends where you're coming from. Go ahead, Andrew.

Andrew
Yeah, So, you know, luckily these these Horcrux cosigner nodes are they're a very lightweight process. And so Horcrux additionally offers a single signer or mode, which is a single node that will hold your validator key, not sharded, but we don't really recommend that for Main Nets. But yeah, you know, that means you have a single additional node that uses very minimal resources.

Andrew
You know, we're talking a single CPU and a gigabyte of ram is all that that the war machine would require. And the Horcrux cosigners are really the same way. So yeah, if you do shard your key now you, you know you need a minimum of three cosigners and that really is our recommended configuration is that you would have three Horcrux Cosigners so you'd have your key sharded into three pieces and with a threshold of two.

Andrew
So that's, that's the configuration that we use on all of our validators and that we recommend that others use as well. And so what that, yeah, that requires three nodes and kind of same resource requirements. There means one CPU, one gigabyte of RAM for each of those machines. That's if you're signing a single chain which up until Horcrux V3 three was all that was supported.

Andrew
So that meant, you know, that you needed a, a Horcrux cluster per chain that you wanted to sign within. Then you would have three nodes each requiring one CPU, one gigabyte of RAM. But now with Horcrux V3, we introduced the ability to sign multiple chains from a single cluster. So what this means is you can load in the Shard for multiple chains and you can point the Horcrux cosigners to the centuries out of those multiple chains and it can sign sign blocks for all of the chains that it's connected to.

Andrew
Yeah. So I mean, a huge improvement because yeah, I mean, right now we're in the middle of the migration as well. We have Horcrux V3 hooked up for a number of our validators, but we're still working to reduce our number of Horcrux clusters because now we really can use a single Horcrux cluster for all of the chains that we signed.

Andrew
So that means that, yes, the resource requirements do go up a little bit on the per cosigner basis, but but it is very minimal. So, you know, if you have a four CPU 16 gigabyte of RAM machine, you can handle, you know, ten, ten or 20 chains.

Citizen Cosmos
Now, you mentioned all the time three signers. No, you didn't say thats the architecture we use. Theoretically, could I shard my key to five pieces. Will I be safer of doing that? More secure or not.

Dan
You should kind of think.

Andrew
Tradeoffs go ahead Dan.

Dan
Oh, yeah. I think I was going the same pace as you like in general. Like if you do like a three or five kind of thing, then you're going to have to have five, Horcrux nodes running, right? So there's a scale out in both complexity from the the infrastructure management and the cost based on like how you're doing it.

Dan
So that'll be that. The biggest thing I'd say it also kind of comes down to like how many people are involved in your business too, because we'll get to this a little bit later. But there's this new mode called DKG where you say that you're in a partnership with like seven people, right? Kind of like I think Noble has like a partnership with like seven people for their ...

Dan
You could and they do like a three of seven or something like that. I might be wrong in the numbers, but yeah, you could do the same thing with a Horcrux so that there's like five partners that you have and you want to run a validator together using DKG. You never share the key. Each of you guys have part of the key material and that'd be like a time you might want to scale up to more than two of three.

Citizen Cosmos
Okay, okay. Now. But what the questions might sometimes seem a little bit like like I said, this is more like, at least in my head that I remember of the things. I honestly didn't write them down and then put them in place. So that might seem like I'm asking them a little bit, but I'll try. I'm going to try and keep it in logical order as you guys answer, of course.

Citizen Cosmos
So the next thing. Right. And I think.

Andrew
That's a great question. I did just want to add one thing to that, which is that, you know, with your total and your threshold configuration, it is a balancing act because it's, you know, what level of availability do you want compared to what level of security do you want? So if you have a threshold of two with a total of three, that means that you can have one cosigner down and continue signing blocks.

Andrew
It also means that you need two of the shards in order to produce a valid signature. If you go up to, you know, like a five total and a threshold of three, that means that you can have two cosigners down. But it does mean that you need, you know, three of the five. So it is a lower percentage than that of the two of three.

Andrew
So you are sacrificing a little bit of the security availability. If you do a four or five. On the other hand, that is more secure than the two of three, but it has the same availability two of three.

Citizen Cosmos
Okay, that makes sense. And relating to that, um, Will Sorry, I had two questions at the same time. Im gonna ask them one by one could theoretically, let's say there is a validator who is, you know, strictly bare metal and you know, they're against having cloud machines and they're part of infrastructure it for Horcrux, it doesn't matter where it runs right?

Citizen Cosmos
I mean, you could run them on a home based computer, right. Any of the signers.

Andrew
Right. Okay. Yeah. The main thing they're trying to think about is the latency between signers. And you have to balance that against, you know, what is the window with which you can submit evidence for signing a block. So if the letancy is too high so you have too many cosigners that are needed to sign a signature and it's a lower block time, you do risk missing blocks.

Andrew
So so you do also need to balance the latency between the machines. Of course, if you're are in a cloud or an environment that has very low latencies, you could have any number of cosigners. And with with no latency, you're going to be able to sign the blocks. No problem now. Yeah, we're we're actually in the middle of our on-premise migration ourselves to cut some of our cloud spend.

Andrew
So yeah, we will be moving our our Horcrux cluster, our Horcrux cosigners to decentralized locations throughout the US.

Citizen Cosmos
Nice. And by the way, this was my next question. So we maybe we can talk a little bit more in detail if you want about that. This was something I think again was Crypto Crew, was it Jacob, I don't remember, to be honest. Sorry, not that I don't, was just really I'm trying to remember all the questions here and the whole block missing thing.

Citizen Cosmos
Now, a lot of validators are terrified of missing blocks. Now, it's a bit silly because, you know, if you have 1% missed blocks or five or whatever the chain is, it can survive without you. But of course not 100%. Right. So. So and then, of course, added Horcrux to your infrastructure will lower the latency so you will miss more blocks now can you reflect on that a little bit say what what's what your thoughts about this.

Andrew
Yeah I mean with our deployments, you know, with the slashing window uptime percentage, we never see it go below 99.9%. And it's typically much higher in the in where, you know, 99.97%, where that means, you know, over the last 16 hours we've missed three blocks potentially. So that's like where we sit that that's in our cloud environment. We do expect that to maybe drop a little bit once we're, you know, Bare Metal and we're using different ISPs for the different locations.

Andrew
But yeah, you know, I would say that when we back back when we had single Node validators, we would see similar percentages of, you know, not always getting 100% in the in a slashing period window. So I think, yeah, you know, it definitely does add latency to the signing process but typically not enough to to miss blocks. And there are there are tuning parameters that you can use time outs that you can tune for the raft consensus and the gRPC timeout so that you know, if a blocksign request between two cosigners is taking too long, it can give up and try again within a shorter time period.

Andrew
So so you come on with it and you know, other validators, we have a telegram channel that happy to invite any other validators that are interested in trying it out, but we can help you, you know, fine tune the operations and minimize latencies too, to make sure that all the blocks are being signed right.

Dan
I want to echo that. That tradeoff that is happening there has a really cool like pro side, too. So even though we're missing three blocks in 16 hours, the pro side is that we have high availability, which means we can have a node go offline so we can like go patch a node with Huckleberry or Burberry or whatever, and we keep signing where if you are running a single sentry without as far as I know, the only high available solution is doing something like Horcrux.

Dan
So if you're just go after 100% uptime with a single sentry and you have to do a patch for some reason without a chain hault, you're going to have a lot of miss locks during that time.

Citizen Cosmos
Make make sense. Make sense. I and what do you by the way but in general sense since you guys are quite big on not missing blocks and I'm sorry that was like sounded like a funny phrase. But what's what's what's what's your opinion on general will on this topic? Sorry to kind of sway aside, but it's not really about missing like one or 2% of blocks.

Citizen Cosmos
Is that acceptable? Is that unacceptable? Is that should validators be afraid of having, you know, a 0.1 to 1% means blocks are what's what's they've been what's your opinion here?

Dan
I've talked to a lot of people about this and a lot of people think that you actually get penalized if you have less than 100% uptime, which is not true at all. You only get penalized if you violate the slashing window. So I think it's kind of dangerous in general to try to pursue 100%. It can lead to mistakes that can involve like Tombstoning your node.

Dan
I think it's much safer every time that I personally do an upgrade. I am on my scan. I'm watching that I'm missing blocks before I bring up the next node, just to make sure that I don't ever risk double signing.

Citizen Cosmos
It's actually smart. And by the way, I think it was also somebody from crypto crew who said an interesting point. If you're if you're seeing a validator with a constant hundred percent performance, probably they're using very lame infrastructure. And because more complex...

Andrew
That may not be the case for it for a well-tuned. Horcrux Yeah, we do have many that do 100% block signs day to day. And yeah you know you'll see a missed block here and there over the course of days. But yeah, that may not be the case. But I will say, you know, if you're seeing in the realm of 1 to 2% missed blocks, there's probably a problem really, even more than half of a percent probably indicates that there's some kind of a problem.

Andrew
But yeah, anything less than that, I don't think there should be anything to worry about There. But even if you have to do a maintenance upgrade on the Horcrux cluster, that means you can't do the rolling upgrade. It's still means You're still missing a, you know, maybe 20 blocks if you're taking all that the signers down, patching the binary and bringing them all back up.

Dan
Yeah. One final thing to say on this too, is like so that slashing parameter that all the different chains like provide like my opinion that's really there for like the smaller validators, the ones that can only afford to run like one century. Like it would kind of suck if they got penalized for like having to reboot the node to apply security patches and not be allowed to have any downtime at all.

Dan
But that slashing parameter allows on almost every single chain any single validator to be down for probably 8 to 10 hours at any point, which allows validators to sleep through the night, when like emergencies happen, which is very nice, and allows them to have time to like, test that their upgrades went smoothly and not have to worry about keeping 100% uptime.

Citizen Cosmos
Okay, that makes sense. And by the way, guys, for the listeners, I mean, but Andrew, I think mentioned the Telegram chat. It's Horcrux have a hangout telegram channel. So I believe this is the chat we're talking about and if you can't find it, go to the Strange Love Telegram chat. And I'm sure from there you will can ask for help and find it.

Citizen Cosmos
So yeah, no stupid question time. Yeah. So one thing about what's going on.

Andrew
We do have a link to the Horcrux Hangout Telegram channel at the end of the V3 blogpost.

Citizen Cosmos
Okay. Nice. Nice. And also, again, if you listen to the recording, you will probably see together with the transcript of the link. But regardless, this is more for the for, for, for the people who are live if you if you having trouble to find it either follow his advice or the lost Telegram chats and you will find it.

Citizen Cosmos
So now stupid Question time guys. Prepare yourself now I warned you. And so and this is again a question that I've been gathering over time. So does that mean and here's the question does this mean and by this I mean somebody using Horcrux that you don't need sentry nodes anymore?

Dan
Yeah, go ahead.

Andrew
Yeah, I was going to let you answer a couple. you go for it.

Dan
Okay. Yeah. So it really comes down to like how you define what a century mode is. So you still need a node that is thinking blocks, right? So the way that it kind of technically works behind the scene is that there's this one setting in conflict called Prove validator and by default is commented out, meaning that you're not using an external remote signer.

Dan
So you still need to have a block that has a genesis file that has like persistent peers, that's connecting to all the other validators and creating blocks, and then you're just connecting that to Horcruxe to that one customization in your config. So you definitely need to still have that part, but you can eliminate the term called Sentry, which I think I don't know who it is coined by, but I know Jack wrote an article about it a few years on a Cosmos blog where he called it a sentry where it was basically a node that sits in front of your other node and through the use of private peering, your node that has the key

Dan
material is never exposed to the public. Only the sensory nodes are exposed to the public and they relay the private information to the private node.

Andrew
So you literally.

Andrew
Cluster that. That is the purpose that they're serving. So it's not that you have like a separate chain binary running that has a print out to you and you have another node in front if that you have the, you know, the Horcrux nodes and another noded product and that other node in front typically is referred to as the sentry node.

Citizen Cosmos
Okay. But then would you again, I'm going to continue on with my line of thought, but would that mean that you would only need one? I mean, I'm going to call it still sentry just for the sake of conversation, but okay, I understand that it's not the best term, but would you still only need one sentry node in front of your Horcrux node or would you add the cluster of them or it doesn't make any sense in this case already.

Andrew
Yeah. So we do clusters of sentrys in front of our Horcrux Cosigners. We typically do end to end. So if we have three Horcrux Cosigners, we'll run three sentrys and we'll connect all of the cosigners to all of the sentrys for more distributed setups. It might make sense to have the cosigners connecting to the nearest sentry, but again, this goes back to the availability tradeoffs like yes, you can function with a single sentry, but then you have a single point of failure.

Andrew
So if that sentry goes down now you have nothing that can sink the chain and send blocksign requests to the Horcrux cluster. So you won't be able to sign as soon as you introduce a second century, then yes, now you have high availability where if one goes down it can continue signing blocks. But yeah, we typically do three centuries that way.

Andrew
You know, we can have as many as two entries go down and this lets us do like rolling updates. This lets us take down a century to do snapshots and then we'll still have two centuries up. So we still have HA during the time that snapshots are taken and those kinds of things too to further solidify the deployment.

Citizen Cosmos
Okay. And now a little bit about the the bad side of Horcrux, what is the bad side? Is there a bad side to Horcrux? But we talked about, you know, the the latency. We talked about the the cost increase especially. Well, it doesn't sound like a lot to me, especially with V3 you guys rolling out. It sounds like pennies, but still, you know.

Citizen Cosmos
Okay, let's take that at the bedside. Are there any other technical or non-technical or economical or any other risks that we are not yet mentioned here today?

Andrew
I mean, it does add complexity to your deployments. And I will say in Horcrux V two like that meant you need a different config file for each cosigner. So there were, you know, a couple of bullet guns in there where like a misconfiguration could mean that like Horcrux was signing blocks, but one of them was misconfigured, so you didn't actually have the high availability.

Andrew
So you know, we've taken a lot of those things into account and like tried to resolve many of those for the V3 release. And so now you can use a single config file for all of the COSIGNERS. So you know, you assembled a config file that will have the definition for all of them and they can all use that same config file.

Andrew
They can identify which cosigner they are from the config and use that. So trying to remove the complexity from the deployment, from the configuration to make it more streamlined and easier to use. And you know, I would say that I don't think there are any make any major reasons not to use Horcrux. I think as soon as you kind of wrap your head around it, as soon as you've tried it, really you'll realize that there's probably no better way yet.

Dan
I can say a great way to just like, try Horcrux out to just jump on a testnet where there's no penalty for double signing. Like it's a that's how I learned Horcrux. I go in there. I tried doing different clusters. Sure, I got double signed in playing around because I forgot to also delete my validator key and I was running two signing machines at one time.

Dan
Yeah, that's how you learn. You got it. That's what test nets are for, is to learn and see how things behave, see how things break and then grow from there.

Citizen Cosmos
Just to clarify, guys, the V3 is out already. Or it's been rolled out.

Andrew
It's out? Yeah, it's been out for two weeks. We just put out a blog post announcement yesterday.

Citizen Cosmos
Okay. Okay. Okay. Okay. Because I did it. Sorry, I didn't understand that part. I was still under the impression the blog post is out, but it's going to be rolled out. Okay, so my apologies there. It's out. So, guys, again, I want another easy way to find, by the way, is go to GitHub and find the release, which is also easy.

Citizen Cosmos
And guys, I'm going to ask a more of a philosophical, I guess, question, the philosophical, but social philosophical. Now, for a perspective of Horcrux, why are people I mean, from what you guys are explained to me today, it sounds well, it's probably even simpler than running a node in a lot of sense because there was a lot of things you don't need to take care of about within a node you would theoretically care about.

Citizen Cosmos
And what is the reason then you think at least this from your perspective of the creators of this thing, of the writters of the code, whatever you want to describe yourself in relation to Horcrux, why are people terrified of this so much? What is the reason?

Dan
I think one factor is that almost all validator documentation out there that exists isn't focused on like Horcrux deployments right through like, Hey, here's how you create your consensus key, here's how you keep track of the state, make sure you protect it, that kind of stuff. So by default, everyone who's like participating in these validator networks, they're just following the official guidance, which does not mention Horcrux, Horcrux is more kind of like a afterthought, or it's like, Oh, I didn't even know there was another way to do it.

Dan
Same thing with TMKMS.

Andrew
Right?

Citizen Cosmos
And by the way, I started going and.

Andrew
Yeah, I'll just say, you know, I think another thing is, you know, maybe you see Horcruxes, you're like, Oh, this, this means that I have, you know, additional nodes I need to manage configuration. And now I need to worry about the networking between those nodes that, that that can seem daunting before you've tried it out. And so I think it's just the port of entry.

Andrew
You know, once you've gotten past that initial understanding and you've tried it out, then I imagine there's there's not a lot of reluctance after that.

Citizen Cosmos
And again, worth saying, I think if you're a little bit familiar with Horcrux or joining in the middle of the conversation, that in the new version, the V3, it's narrowed down now to one cluster of Horcrux nodes. So the networking again goes minimizes the amount of networking you will have to do. It's not like the cluster for each of the, let's say, ten networks, you are validating.

Citizen Cosmos
And so, yeah, I was going to ask about the comparison to. Dan, you mentioned the ... a couple of times, Um, do you have any presence or any pros and cons if you put them against each other or something?

Dan
So it's been a little while since I've used TMKMS, but I'll go based off of my memory, my personal experience with it. So I started off with a soft sign like default thing, and then I learned about TMKMS as a remote signer, so I switched to that. They support a couple different methods. They support a soft sign as well, which is basically no additional security as far as I can tell.

Dan
It's just it decouples your signer from the the block producer or the block indexer. The second thing that they have is they have a HSM integration. So you can buy a device called the Cube HCM. And if you have a physical server that you can stick a device into, which is hard when you're doing like cloud computing, but if you're running bare metal, it's very easy, then you have a very, very secure signing solution.

Dan
The key is on that device just almost like a ledger. The key doesn't come off. Maybe that's a bad comparison, But, you know, I mean, it's a hardware security module, so they're made to prevent the key material from being extracted. And that's one advantage that TMKMS has that I don't know of any others that currently support that.

Dan
But what they don't have is that as far as I know, it's a high availability solution. If you're using TMKMS, you're only having one signer machine. So if you're a signer goes down for whatever reason, you're going to stop signing blocks.

Citizen Cosmos
Can I ask you something here? You said one sign our machine, as far as I'm aware with yubikeys you can double them as in like, you can repeat them so you could replicate the same yubikey. Like, for example, you know, when you can use it, when you log into a computer and when you do it, you should really put not one but two.

Citizen Cosmos
So in this case, would I still have one signer machine if you describe it, or I would have two, because I'm like...

Dan
That's a good point and I don't know the exact answer, but if you did have the same key on two different signers, you need to make sure that a double sign doesn't happen. So I don't know if TMKMS, supports that, but that's one of the things that Horcrux supports is it keeps track of. It is a node sign if it did absolutely prevent any other node from signing.

Dan
So you'd have to have some sort of tracking mechanism for that.

Citizen Cosmos
Okay. Just a note to to everybody listening. And if you do use a yubikey now from personal pain guys, if you're using it for your personal computer or your transaction or personal computer, whatever, get two you will save yourself a lot of a lot of crying. I mean, there are solutions, but it's better to get through. This is about personal computing.

Citizen Cosmos
Your guys, whoever is listening to this note, nothing to do with Horcrux. So yeah, sorry. But back to back to Horcrux. Now, you mentioned at the beginning, I don't remember if it was Dan you I think it was about talking a little bit down the line about something in particular with regards to what you guys have been either developing or the workshop you guys wanted to talk about.

Citizen Cosmos
Do you want to mention it?

Dan
Sure. There's no set date yet, but we're we have a relationship with Dora Hacks and they host a hackathons for all different types of blockchains. One of them is Cosmos, and they are kind of approached us and thought, Hey, it's kind of cool if you guys did a workshop on some of the technology that you guys have built.

Dan
So we kind of threw out Horcrux as the first idea. So once we have our details on that, I'll definitely tweet out some information and probably it'll be from the Strangelove account. So if you want to sign up and do Horcrux workshop, feel free to attend.

Citizen Cosmos
I think you should, guys. Everybody was listening to this. My advice, it sounds like it sounds like it solves a lot of a lot a lot of issues, psychological issues for the operator. Let's put it like this. It's at least at the very least it does. There's a lot more from what it sounds like. Okay, I'm now again back to Horcrux.

Citizen Cosmos
What else? I didn't ask you, which is from the obvious things, because like I said, I did admit the questions were coming from the top of my head and trying to remember what people have asked me on on different occasions and what I had in my own head. Now, is there something that I didn't ask in relation to like a higher overview of Horcrux?

Andrew
I think we covered a good chunk of it, you touched on what is next, like what's what's coming. So, you know, to talk about the roadmap a little bit, Dan mentioned one of the things that we're excited to to release, which is DKG key creation. So like he like you alluded to, this means that rather than bringing your ... JSON and sharding it into multiple pieces, you actually have multiple parties come together and declare, you know, what is my cosigner ID want and, and participate in this ceremony that creates a public key from the shards.

Andrew
So then after that ceremony is completed, you have your private key shard. But during that ceremony, there was no secret material exchanged. So you end up with multiple parties that were able to create a public key that can be registered for the validator without requiring any of the parties to to share their secrets. So that's something that we're very excited to roll out, you know, daos and other things where you where you have multiple parties that want to participate in a validator.

Andrew
This will be huge for those applications. Another thing is, so Rafael, who's on the call here, he actually did a great refactor for us to abstracts the local cosigner to open us up to NHSM integration. So similar to the TMKMS, how they offer like a yubikey HSM integration. We would like to do the same thing with Horcrux where rather than having your full key, your full ID 205.9 private key on the YUBIKEY, you would actually just have your key shard and you'd be able to use the Yubikey or other integrations such as, you know, Google Cloud or AWS as their HCM cloud solutions to plug those in to, you know, further add to the

Andrew
security of Horcrux you know, this is especially applicable to us now as we're moving to an on premise deployment where we will have our our Horcrux Cosigners put down in a physical location that that we want. You know, we're looking at co-location setups. So very important to us to keep our shards secure with the hardware that's deployed in a location where we're renting Rackspace potentially.

Andrew
So yeah, that's something that I'm very excited about to further secure Horcrux. So that way you have the the added benefit of a private security module where your key never leaves the device. All the signing happens on that device and also the high availability that Horcrux offers.

Citizen Cosmos
And now I'm going to be devil's advocate here, but let's just follow follow with me. It's good intentions and now it sounds when you talk about the roadmap, like the security is getting more and more complex and is that not going to scare the people away? I said I'm going to be devil's advocate. I did warn you. So please, what do you say?

Andrew
And that's where I think, you know, it won't be a required configuration. It won't mean that you're required to use an HSM to use Horcrux, but it will be an additional option that's available to you if you do have those needs. And so, yeah, you may not have those needs. If you if you have the servers in your own locations that you own, where you know that the key shards are are under lock and key physical lock and key as well as by Horcrux.

Andrew
But, you know, there might be other deployments that would benefit from this strategy. Yeah, we definitely don't want to add complexity that's forced on Horcrux operators. But, you know, in the name of additional security, additional availability and those kinds of things, we definitely would like to add those those features to improve the operator experience for advanced for the power users of Horcrux.

Dan
And I'll say that most of the users out there are it's suffice for them to use like the default kind of like best practice config that most people use and that's literally only like 20 lines of YAML. It's just one file that has configuration where you say, Hey, these are my keys, these are the the nodes that run the different blockchains and that's it.

Dan
And Horcrux, does the rest.

Citizen Cosmos
This is my keys and this is my beer and Horcrux does the rest of it. Now, one more devil's advocate question, guys. Now, Horcrux is a product like, you know, I mean, let's separate this for a second from strange love and from everything else. Just just to make my little question now, does that mean that in the future and of course, now you guys are talking about a roadmap already, which probably means that, you know, when you mentioned Rafael, which means that there are probably other engines.

Citizen Cosmos
Anyway, long story short, will we get to a point where Horcrux will start charging for additional features or it will always stay for now at least an open source product, which is available to anybody to use?

Andrew
Oh, absolutely. Horcrux will always be open source, you know, with that Apache two license on there. We wouldn't do anything like that, like make it make it closed source and charge people to use it like a licensing fee or anything like that. We do productize most of our open source software at strange love. So, you know, we do have a validated product where you know, white label validator product where we can run the validator for you and you can manage the account key, we'll manage the consensus key so that you know, the customer then can, can manage the funds, the claiming, the staking rewards and all of the custody operations.

Andrew
And we'll just run the the Horcrux validator infrastructure. But yeah, we, you know, that's typically our play with our open source software. So we're not we definitely do not want to make it any more code closed source. We want it to be available for anyone to use because, you know, then we get the benefit of having external contributors point out places where we can improve our code.

Andrew
So we definitely to keep that. Yeah, that's the track where the Strange Lives motto there.

Citizen Cosmos
Okay. And Dan, did you want to add something?

Dan
I no I mean Andrew is really the best one to speak on this as the, the director of engineering. He's got a lot of influence on how the products are mature but yeah like he said, it's open source so it's always going to out there and available. Even if whatever reason the repo went away, someone could always fork it and continue the project.

Citizen Cosmos
Right. And and just, just to add to what kind of to add a little bit of a side note for anybody listening and you know if you for a second wondered I mean probably didn't but if there is a non validators presence which I actually see some and you're just curious you know about how can about the keys management so yeah so Tendermint offers what's called permissions and grants and you can actually hierarchies accounts and you can get permissions for withdrawal commissions for for un-jailing for, for voting and so on and so forth.

Citizen Cosmos
So that's actually great. It's another, by the way, layer of security, I guess, in a way. So yeah, that's what's Andrew what it was referring to. And guys, do you want to add anything else? It feels like, like, you know, like, like it was I had this expectation of like, oh my God, okay, this is a complicated everybody's going to get confused.

Citizen Cosmos
Like, but it's not. It just really simple. So like, well, I'm sorry to simplify, oversimplify it, but, but it is in a sense, and I don't really like I had some other questions, but you kind of already answered them. So if one first of all, by the way, of course, if anybody listening right now wants to ask the question like once again, if you are to shy DM it to me, I'm in front of a screen, I should see it or you can raise your voice and I will just let you speak or raise your hand so much as your voice and you can try.

Citizen Cosmos
And that probably won't work. But yeah, I either. And then, sorry, I'm going to get back to Andrew. And do you want to add anything? I didn't know. Not even about high overview. Let let's maybe, you know, anything else to add for somebody who wants to know, try it out. Apart from trying it out on this Testnet.

Andrew
Yeah. I'm not sure if there's much more I could add other than just say, you know, I'm really excited about this to be V3 release. Great for anyone to go. Go try it out, you know, give us feedback. We love all of the feedback we get from the community. It helps us improve the project. And yeah, it's great to work with with all of you and help help improve the Cosmos ecosystem overall.

Citizen Cosmos
Awesome. Dan. Any last last words. Sorry.

Dan
Yeah, just kind of echoing. Andrew said. I just want to say come join the workshop if you want to get some hands on with it or join our telegram group. We're very active in there on helping people to understand how Horcrux can help secure the validator guys.

Citizen Cosmos
So thank you very much. Just again, and a couple of resonance for me from what the guys said. There is the new version, which has a lot of improvements, especially in terms of what it will add to your costs and the complexity. What it will not add to your complexity or other is out. So go check it out.

Citizen Cosmos
There's an article that Andrew mentioned that describes it, and as Dan said, the best way to do it is just to start using it on a testnet. and them go and double sign or whatever. Well, you probably should actually, on a test, do it on purpose, in my opinion, so you understand how it works if you're a lot of data and don't be afraid of of playing around with it.

Citizen Cosmos
Yeah. And definitely I personally look a lot forward. I know one of my DevOps team is on this call right now listening and I'm looking forward to slowly starting implementing that as well. And especially now with that I did know about the cluster thing in the V3 that's changes in my opinion, a lot of things. So especially for smaller validators like ourselves, you know.

Citizen Cosmos
So yeah, yeah, that's it from my side I guess. Guys, if anybody else has questions, you still have time to ask them, but I'm not so enhance. So. Yeah. Okay guys, then watch out for the news from Strange love about the workshop. It will be, as I mentioned, I think it was done right on the Twitter of stranger love. And Dan and Andrew, thank you very much, guys.

Dan
Thanks for having us.

Citizen Cosmos
Thank you all game. Okay. Thanks, you guys. Thank you for listening, people. Thanks. And start using Horcrux. Okay. Bye, everyone.

If you would like to support our mission in creating educational content and aligning the goals of different communities, please stake with us here:

Join our community, to build a future where communication is decentralized. May the code be with you!