Pete's Log: Doing Things and more iSCSI woes

Entry #2003, (Coding, Hacking, & CS stuff)
(posted when I was 43 years old.)

Until recently, my 5-pi-cluster (still looking for a good name for it) has looked as follows:

  • Unused
  • Unused
  • Bastion Host (I just recently learned this term and I like it)
  • Prometheus and Grafana
  • Home Assistant (misbehaving)

I've spent a lot of time reading and pondering and hemming and hawing about how to build a more clustery cluster. I finally decided that being able to work on this only in small bursts, I have to just pick something and go and break things and maybe learn things.

So the two previously unused pis are now a prototype cluster. I haven't gotten terribly far because I had to spend a lot of time fighting iSCSI, but last night I finally made a few breakthroughs. My plan is to figure out half-decent ways of getting the apps working that I want working, take a lot of good notes, and then tear it down and build it up cleaner. Which has proven very refreshing, because I can just liberally run kubectl apply -f some.yml and not stress too much about the results.

So here's what I have so far:

  • Two pis running Ubuntu
  • K3s Kubernetes in a basic one master/one worker configuration
  • Longhorn for storage management
  • Functioning Mosquitto and ESPHome deployments

It's overkill, but it's exciting to see something coming together. Next steps:

  • Create deployments for
    • Home Assistant
    • Prometheus
    • Grafana
    • Home Assistant Add-Ons like zwave-js and network ups tools
  • Maybe add another pi or two to the cluster (long term I think all of them except the bastion host will be part of it)
  • See if I want to use something like Flux
  • Keep reminding myself not to think about it too much and instead to keep Doing Things

I've gone back and forth a bunch about should I be absurd and use Kubernetes? Or just Docker? Or give the HA OS another chance? It was working real well for a while, after all... Maybe I should just give into its demands and let it have unrestricted DNS access to the internet.

Nah, absurdity it is.

I'd love to have a setup where if a particular pi dies, I can just replace it with minimal fuss. I'm not sure if this is going to get me there. But maybe that's OK. Certain things will still be tied to a specific pi, since the zwave stick and the UPS USB cable need to plug in somewhere. And a high-availability setup would require three master nodes, leaving just one worker node. I'm getting rambly and it's past my bedtime. So I'll just leave this here:

I really need to get better at reading comprehension. At the top of /etc/iscsi/iscsid.conf it says the following:

# # Open-iSCSI default configuration. # Could be located at /etc/iscsi/iscsid.conf or ~/.iscsid.conf # # Note: To set any of these values for a specific node/session run # the iscsiadm --mode node --op command for the value. See the README # and man page for iscsiadm for details on the --op command. #

So if you add a node and then afterwards change any settings in that file, those settings changes won't apply to the existing node. Which is actually pretty great. Just frustrating when you don't read closely the first couple times editing the file. So as a reminder to myself and also since the internet seems to keep leading me astray on iSCSI topics, I'll just store this here.

See the current settings for a node:

iscsiadm -m node -T iqn.to.query

Update a setting:

iscsiadm -m node -o update --name setting.name --value value

Examples:

iscsiadm -m node -o update --name node.startup --value automatic iscsiadm -m node -o update --name node.session.auth.username --value username iscsiadm -m node -o update --name node.session.auth.password --value password