[{"data":1,"prerenderedAt":2120},["ShallowReactive",2],{"indulge-series-posts":3},[4,163,525,542,564,1359,1644,1754,1813,1830,2020,2065],{"id":5,"title":6,"body":7,"coverImage":143,"date":144,"description":145,"draft":146,"extension":147,"featured":146,"group":148,"initial":146,"meta":149,"navigation":150,"path":151,"rank":152,"rawbody":153,"readTime":154,"seo":155,"series":150,"seriesCover":146,"seriesOrder":152,"seriesSlug":152,"slug":152,"stem":156,"subtitle":157,"tags":158,"__hash__":162},"indulge/indulge/business-minded-development.md","Business-driven Development",{"type":8,"value":9,"toc":135},"minimark",[10,14,17,24,27,30,36,39,44,55,73,76,79,83,86,89,92,95,98,101,105,108,111,114,123,126,129,132],[11,12,13],"p",{},"What if your multi-tenant application could be shipped as a distributed SaaS? What if you could have a licensed version of your product as an installable on some corner of the internet? What if we could break the barrier of what it meant to be a revenue generating company? Let's begin.",[11,15,16],{},"Software is the means by which businesses achieve mass distribution. Beyond technical acumen and efficiency, software is the bus through which businesses achieve scale.",[11,18,19,20],{},"Over the quarter, I have been working on software distribution. Specifically: licensing. Think of it in terms of introducing a new avenue of revenue. We dare ask: Do the businesses that ",[21,22,23],"em",{},"value this solution want control over where their data lives? Are they willing to pay a premium over this control?",[11,25,26],{},"The answer, in both cases, was yes and that opened a door I had not thought to knock on before.",[11,28,29],{},"I had to stop thinking about features and lean towards markets and distribution surfaces. Distribution, in this sense, is not about deployments and pipelines. It is about understanding that different buyers have fundamentally different relationships with trust, compliance, and infrastructure ownership. Your architecture either accommodates those differences or silently forecloses them.",[11,31,32,33],{},"Being technical helped me deploy features product needed. Being business-minded led me to more questions. ",[21,34,35],{},"What would it take to ship this as a product someone else operates? What are we assuming about shared infrastructure that a regulated industry, a government agency, or a security-conscious enterprise would never accept?",[11,37,38],{},"The answers to those questions are not found in a ticket or a sprint. They surface when you sit long enough with the business model to let it ask questions back at you.",[40,41,43],"h3",{"id":42},"licensing-as-a-pricing-model","Licensing as a pricing model",[11,45,46,47,54],{},"For ",[48,49,53],"a",{"href":50,"rel":51},"https://navengine.encha.cloud",[52],"nofollow","Navengine",", the questions came back as licensing. Once you accept that a buyer might want to run your software in their own environment - air-gapped, on-premise, inside a VPS they control - you need a mechanism that lets the software know it is legitimate without phoning home in ways that violate their security posture. You need entitlements, activation flows, machine-bound licenses and expiry.",[56,57,63],"div",{"className":58},[59,60,61,62],"max-w-2xl","mx-auto","text-center","text-xs",[11,64,65,70],{},[66,67],"img",{"alt":68,"src":69},"forbidden","/business-dev/forbidden-problems.webp",[21,71,72],{},"It turns out the problem I was solving was not purely technical.",[11,74,75],{},"Somewhere in the middle of designing the activation flow, I realised the license schema was not a technical artifact. It was a pricing model. The fields I was choosing - seats, features, expiry dates, activation limits - were decisions about how we intended to capture value, just expressed as database columns. It was literally us deciding how we wanted to make money, written in schema form. And no product manager or founder had handed me a neat spec sheet. I was the one typing those business decisions into code.",[11,77,78],{},"That kind of realisation doesn’t land with a dramatic “aha!” moment and applause. It creeps in quietly. You’re just fiddling with fields one afternoon… and then, almost without noticing, the way you look at your own codebase has already changed. You start reading the same migrations and models like they’re business strategy documents instead of “just” backend plumbing.",[40,80,82],{"id":81},"the-second-order-effects-arrive","The second-order effects arrive",[11,84,85],{},"When you decide to support self-hosted deployment, you are not making a DevOps decision. You are ratifying a set of architectural choices that were either made deliberately or inherited by accident. Configuration that was hardcoded because \"we only have one environment\" now needs to be externalized. Database assumptions that worked fine in a shared-infrastructure model now need isolation boundaries. Upgrade paths that were previously your problem - a migration you ran on a schedule you controlled - become your customer's problem, which means they become your support problem, which means they were always your design problem.",[11,87,88],{},"The business-minded engineer understands this before the contract is signed. They look at a feature and ask not just \"how do we build this\" but \"how does this behave when someone else is operating it?\" That is a different question and it produces different code. It produces code that fails loudly with actionable errors instead of silently with corrupted state. It produces configuration schema that are documented because they have to be, not because someone found time. It produces container images that carry their dependencies rather than assuming a shared environment that may not exist on the other side of a firewall.",[11,90,91],{},"Containerizing Navengine using Flatcar images was not primarily a reliability choice - it was a distribution choice. The image had to be self-contained because I could not know what the environment on the other side looked like. Keygen had to be reachable at activation but not required at runtime, because an installation behind a firewall cannot tolerate a dependency on an external service it may never reach. None of that came from an architecture review. It came from thinking hard about the buyer and working backwards from their constraints.",[11,93,94],{},"That backward motion - from buyer to architecture - is the discipline that business-minded engineers practice. It is less intuitive than building forward from requirements, and it is rarely taught, because most engineering education treats the business as a context for the technical problem rather than as a force that shapes the technical solution. But the shape of a system tells you exactly what assumptions its builders made about how it would be used, and by whom, and under what conditions.",[11,96,97],{},"If Navengine could only really be operated, understood, or debugged by the people who built it… it would cease to be a product. It would be a service that’s been dressed up in product clothing.",[11,99,100],{},"The distinction matters because services and products scale differently. A service scales with people - support engineers, customer success managers, implementation specialists. A product scales with copies. When you ship a licensed, self-hosted deployment, you are betting that the copy can stand on its own. That bet is won or lost in the architecture, long before the first installation.",[40,102,104],{"id":103},"none-of-this-is-free","None of this is free.",[11,106,107],{},"I will not pretend this is clean. Supporting two distribution surfaces - a managed SaaS and a self-hosted licensed product - means every feature has to answer for itself in both worlds. Every abstraction either holds across deployment models or eventually creates a seam you will debug at the worst possible time. The surface area grows and so does the discipline required to keep it coherent.",[11,109,110],{},"This is why most teams never do it. Not because it is technically beyond them, but because it requires a kind of thinking that does not come naturally when you are moving fast and the backlog is full. It requires you to slow down and ask what you are actually building - not in terms of features, but in terms of the thing a customer would take ownership of, operate independently, and stake their own reputation on. That is a different standard than passing your test suite. It is a harder one.",[11,112,113],{},"But it is also where engineering becomes interesting in a way that pure technical depth rarely matches. When you understand the business well enough to make architectural decisions that preserve future revenue options, you are no longer just implementing someone else's vision. You are shaping the product's possible futures. That is leverage of a different kind - not the leverage of writing faster code or designing more elegant abstractions, but the leverage of making choices today that compound into distribution tomorrow.",[11,115,116,117,122],{},"John Carmack once said the most important thing an engineer can do is ",[48,118,121],{"href":119,"rel":120},"https://x.com/ID_AA_Carmack/status/1637087219591659520",[52],"build full product skills",". I used to read that as advice about career positioning. Working through this, I think it means something more specific: learn to see the economic system your technical system lives in. Features are temporary. Architectural commitments are long. A business model encoded in a data structure outlasts the engineer who designed it.",[11,124,125],{},"The business-minded engineer does not stop caring about the craft. If anything, they care more - because they understand what the craft is in service of. Every clean abstraction is a future operator's sanity. Every well-chosen failure mode is a support ticket that never gets filed. Every deployment boundary that was thought through carefully is a market that remains reachable.",[11,127,128],{},"Software is the means by which businesses achieve mass distribution. You already knew that. The question is whether the software you are writing is ready to be distributed - not just deployed, not just maintained, but handed to someone else, in a corner of the internet you will never see, running a business you will never fully understand, solving a problem you helped make solvable.",[11,130,131],{},"Becoming a business-minded engineer doesn't mean every engineer should be writing pricing decks. It means understanding that every architectural choice is also a business choice - and that some of the most impactful technical work you'll do is quietly expanding the distribution surface of the thing you're building.",[11,133,134],{},"That is the bar. Build to it.",{"title":136,"searchDepth":137,"depth":137,"links":138},"",2,[139,141,142],{"id":42,"depth":140,"text":43},3,{"id":81,"depth":140,"text":82},{"id":103,"depth":140,"text":104},"business-dev/adding-more-people-to-a-project.webp","2026-03-01T11:58:15.409Z","The mindset shift - from builder to distributor.",false,"md","Distributed Systems Field Notes",{},true,"/indulge/business-minded-development",null,"---\ntitle: Business-driven Development\ncoverImage: business-dev/adding-more-people-to-a-project.webp\ndate: 2026-03-01T11:58:15.409Z\ndescription: The mindset shift - from builder to distributor.\ndraft: false\nfeatured: false\ngroup: Distributed Systems Field Notes\ninitial: false\nreadTime: 8\nseries: true\nseriesCover: false\nsubtitle: What licensing Navengine taught me about distribution\ntags:\n  - distributed systems\n  - product engineering\n  - infrastructure as code\n---\n\nWhat if your multi-tenant application could be shipped as a distributed SaaS? What if you could have a licensed version of your product as an installable on some corner of the internet? What if we could break the barrier of what it meant to be a revenue generating company? Let's begin.\n\nSoftware is the means by which businesses achieve mass distribution. Beyond technical acumen and efficiency, software is the bus through which businesses achieve scale.\n\nOver the quarter, I have been working on software distribution. Specifically: licensing. Think of it in terms of introducing a new avenue of revenue. We dare ask: Do the businesses that *value this solution want control over where their data lives? Are they willing to pay a premium over this control?*\n\nThe answer, in both cases, was yes and that opened a door I had not thought to knock on before.\n\nI had to stop thinking about features and lean towards markets and distribution surfaces. Distribution, in this sense, is not about deployments and pipelines. It is about understanding that different buyers have fundamentally different relationships with trust, compliance, and infrastructure ownership. Your architecture either accommodates those differences or silently forecloses them.\n\nBeing technical helped me deploy features product needed. Being business-minded led me to more questions. _What would it take to ship this as a product someone else operates? What are we assuming about shared infrastructure that a regulated industry, a government agency, or a security-conscious enterprise would never accept?_\n\nThe answers to those questions are not found in a ticket or a sprint. They surface when you sit long enough with the business model to let it ask questions back at you.\n\n### Licensing as a pricing model\n\nFor [Navengine](https://navengine.encha.cloud), the questions came back as licensing. Once you accept that a buyer might want to run your software in their own environment - air-gapped, on-premise, inside a VPS they control - you need a mechanism that lets the software know it is legitimate without phoning home in ways that violate their security posture. You need entitlements, activation flows, machine-bound licenses and expiry.\n\n::div{.max-w-2xl.mx-auto.text-center.text-xs}\n![forbidden](/business-dev/forbidden-problems.webp)_It turns out the problem I was solving was not purely technical._\n::\n\nSomewhere in the middle of designing the activation flow, I realised the license schema was not a technical artifact. It was a pricing model. The fields I was choosing - seats, features, expiry dates, activation limits - were decisions about how we intended to capture value, just expressed as database columns. It was literally us deciding how we wanted to make money, written in schema form. And no product manager or founder had handed me a neat spec sheet. I was the one typing those business decisions into code.\n\nThat kind of realisation doesn’t land with a dramatic “aha!” moment and applause. It creeps in quietly. You’re just fiddling with fields one afternoon… and then, almost without noticing, the way you look at your own codebase has already changed. You start reading the same migrations and models like they’re business strategy documents instead of “just” backend plumbing.\n\n### The second-order effects arrive\n\nWhen you decide to support self-hosted deployment, you are not making a DevOps decision. You are ratifying a set of architectural choices that were either made deliberately or inherited by accident. Configuration that was hardcoded because \"we only have one environment\" now needs to be externalized. Database assumptions that worked fine in a shared-infrastructure model now need isolation boundaries. Upgrade paths that were previously your problem - a migration you ran on a schedule you controlled - become your customer's problem, which means they become your support problem, which means they were always your design problem.\n\nThe business-minded engineer understands this before the contract is signed. They look at a feature and ask not just \"how do we build this\" but \"how does this behave when someone else is operating it?\" That is a different question and it produces different code. It produces code that fails loudly with actionable errors instead of silently with corrupted state. It produces configuration schema that are documented because they have to be, not because someone found time. It produces container images that carry their dependencies rather than assuming a shared environment that may not exist on the other side of a firewall.\n\nContainerizing Navengine using Flatcar images was not primarily a reliability choice - it was a distribution choice. The image had to be self-contained because I could not know what the environment on the other side looked like. Keygen had to be reachable at activation but not required at runtime, because an installation behind a firewall cannot tolerate a dependency on an external service it may never reach. None of that came from an architecture review. It came from thinking hard about the buyer and working backwards from their constraints.\n\nThat backward motion - from buyer to architecture - is the discipline that business-minded engineers practice. It is less intuitive than building forward from requirements, and it is rarely taught, because most engineering education treats the business as a context for the technical problem rather than as a force that shapes the technical solution. But the shape of a system tells you exactly what assumptions its builders made about how it would be used, and by whom, and under what conditions.\n\nIf Navengine could only really be operated, understood, or debugged by the people who built it… it would cease to be a product. It would be a service that’s been dressed up in product clothing.\n\nThe distinction matters because services and products scale differently. A service scales with people - support engineers, customer success managers, implementation specialists. A product scales with copies. When you ship a licensed, self-hosted deployment, you are betting that the copy can stand on its own. That bet is won or lost in the architecture, long before the first installation.\n\n### None of this is free.\n\nI will not pretend this is clean. Supporting two distribution surfaces - a managed SaaS and a self-hosted licensed product - means every feature has to answer for itself in both worlds. Every abstraction either holds across deployment models or eventually creates a seam you will debug at the worst possible time. The surface area grows and so does the discipline required to keep it coherent.\n\nThis is why most teams never do it. Not because it is technically beyond them, but because it requires a kind of thinking that does not come naturally when you are moving fast and the backlog is full. It requires you to slow down and ask what you are actually building - not in terms of features, but in terms of the thing a customer would take ownership of, operate independently, and stake their own reputation on. That is a different standard than passing your test suite. It is a harder one.\n\nBut it is also where engineering becomes interesting in a way that pure technical depth rarely matches. When you understand the business well enough to make architectural decisions that preserve future revenue options, you are no longer just implementing someone else's vision. You are shaping the product's possible futures. That is leverage of a different kind - not the leverage of writing faster code or designing more elegant abstractions, but the leverage of making choices today that compound into distribution tomorrow.\n\nJohn Carmack once said the most important thing an engineer can do is [build full product skills](https://x.com/ID_AA_Carmack/status/1637087219591659520). I used to read that as advice about career positioning. Working through this, I think it means something more specific: learn to see the economic system your technical system lives in. Features are temporary. Architectural commitments are long. A business model encoded in a data structure outlasts the engineer who designed it.\n\nThe business-minded engineer does not stop caring about the craft. If anything, they care more - because they understand what the craft is in service of. Every clean abstraction is a future operator's sanity. Every well-chosen failure mode is a support ticket that never gets filed. Every deployment boundary that was thought through carefully is a market that remains reachable.\n\nSoftware is the means by which businesses achieve mass distribution. You already knew that. The question is whether the software you are writing is ready to be distributed - not just deployed, not just maintained, but handed to someone else, in a corner of the internet you will never see, running a business you will never fully understand, solving a problem you helped make solvable.\n\nBecoming a business-minded engineer doesn't mean every engineer should be writing pricing decks. It means understanding that every architectural choice is also a business choice - and that some of the most impactful technical work you'll do is quietly expanding the distribution surface of the thing you're building.\n\nThat is the bar. Build to it.\n",8,{"title":6,"description":145},"indulge/business-minded-development","What licensing Navengine taught me about distribution",[159,160,161],"distributed systems","product engineering","infrastructure as code","JADQXn4Ru04iW8mYPFpjl2GYzV5ek-Ty0WGyxiIkJc8",{"id":164,"title":165,"body":166,"coverImage":509,"date":510,"description":511,"draft":146,"extension":147,"featured":146,"group":148,"initial":146,"meta":512,"navigation":150,"path":513,"rank":152,"rawbody":514,"readTime":154,"seo":515,"series":150,"seriesCover":146,"seriesOrder":152,"seriesSlug":152,"slug":152,"stem":517,"subtitle":518,"tags":519,"__hash__":524},"indulge/indulge/continuous-integration-for-licensed-software.md","CI/CD for Licensed Software You Don't Host",{"type":8,"value":167,"toc":500},[168,183,190,193,196,200,203,206,209,211,224,226,230,233,252,255,257,261,264,278,291,300,303,305,314,316,320,332,335,348,351,354,357,359,363,368,379,382,385,388,390,399,401,405,408,411,414,416,421,437,450,452,461,463,467,470,476,479,488,491,494,497],[11,169,170,171,176,177,182],{},"We are three weeks away from shipping ",[48,172,175],{"href":173,"rel":174},"https://navengine.encha.cloud/",[52],"NavEngine v4",", an echo from the previous piece on ",[48,178,181],{"href":179,"rel":180},"https://www.marvinkweyu.net/indulge/business-minded-development",[52],"Business Driven development",".",[11,184,185,186,189],{},"I say \"",[21,187,188],{},"shipping","\" loosely. There is no deployment script I run, no SSH session I open and no Kubernetes rollout I watch. The software lives on infrastructure I have never seen, behind firewalls I cannot reach, on machines whose specs I do not know. Shipping, in this context, means pushing an image to a registry and trusting that a process running inside a container on a customer's server will eventually notice and do something about it.",[11,191,192],{},"That gap - between what I push and what the customer runs - is what this piece is about.",[194,195],"hr",{},[40,197,199],{"id":198},"the-assumption-collapse","The Assumption Collapse",[11,201,202],{},"Every CI/CD tool I have ever used was built on a premise so foundational that nobody thought to state it: you control the deployment target. You own the server. You have the keys. Deployment, in the conventional sense, is just automation wrapped around access you already have.",[11,204,205],{},"NavEngine is quite the opposite. It exists as a custom image - qcow2 - shipped onto the customer's infrastructure. The customer owns the machine. I do not have SSH access unless I go through DWService and even then, that is a support channel, not a deployment one. Yet somehow, we need to continuously deliver updates to machines we cannot reach, across connections we cannot guarantee, without breaking software that is actively in use.",[11,207,208],{},"So the question became: if you cannot push, how do you deliver?",[194,210],{},[212,213,214],"blockquote",{},[11,215,216,220,221],{},[217,218,219],"strong",{},"Indulge"," ",[21,222,223],{},"The moment you decide to ship software you don't host, you have made a decision with consequences that will follow you for the life of the product. Not just operationally. Architecturally. Every assumption your codebase makes about the environment it runs in now belongs to someone else's infrastructure. That is not a deployment problem. It is a design problem that shows up at deployment time.",[194,225],{},[40,227,229],{"id":228},"the-answer-is-pull","The Answer is Pull",[11,231,232],{},"Watchtower. It runs as a container alongside the rest of the stack, polls the Docker registry on a configured interval, and when it detects a new image digest on the tag it is watching, it pulls and restarts the relevant containers. No webhook, no push, no SSH. The installation phones home for updates and takes what it finds.",[11,234,235,236,240,241,244,245,248,249,251],{},"The key design decision here was the floating tag. Every customer's Butane config ships with ",[237,238,239],"code",{},"core:stable",". Not ",[237,242,243],{},"core:v3.0.39",". Not a digest pin. ",[237,246,247],{},"stable",". When Watchtower polls the registry and sees that ",[237,250,247],{}," now resolves to a different digest than what is currently running, it pulls. What \"stable\" points to is entirely under my control, from the registry side, without touching anything on the customer's machine.",[11,253,254],{},"This sounds obvious once you say it. It took longer than I would like to admit to get there.",[194,256],{},[40,258,260],{"id":259},"two-registries-two-floating-tags-one-gate","Two Registries, Two floating tags, One gate",[11,262,263],{},"Here is the full pipeline as it actually runs.",[11,265,266,267,270,271,274,275,277],{},"Every push to ",[237,268,269],{},"main"," triggers a build on the dev registry. The image gets tagged with a version identifier and a floating tag - ",[237,272,273],{},"staging",". A staging environment - running the same Compose stack, same Butane configuration, same structure as a customer installation - pulls from ",[237,276,273],{},". This is where the image lives until I am satisfied it works.",[11,279,280,281,284,285,287,288,290],{},"When staging looks good, I create a release. The production registry builds from that release, tags the image with the version (",[237,282,283],{},"v3.0.40",") and overwrites the floating ",[237,286,247],{}," tag. Customer installations, on their next Watchtower poll interval, see a new digest on ",[237,289,247],{}," and update.",[11,292,293,294,296,297,299],{},"The critical detail: ",[237,295,247],{}," is never overwritten by a push to ",[237,298,269],{},". Only by a release. The staging gate is the only thing standing between a broken image and a customer's running installation. There is no automated rollout percentage, no canary fleet, no gradual traffic shift. The gate is a human decision, made after watching staging run and deciding it is ready.",[11,301,302],{},"For a solo-operated product at this stage, that is the right call. Complexity in release infrastructure that you do not need is just surface area for things to go wrong.",[194,304],{},[212,306,307],{},[11,308,309,220,311],{},[217,310,219],{},[21,312,313],{},"A staging environment that does not accurately reflect production is a very expensive placebo. It gives you confidence without giving you information. The hardest thing about shipping self-hosted software is that your staging environment runs on infrastructure you understand, with data you created, on a network you control. Your customer's environment is none of those things. No pipeline fully closes that gap. The best you can do is know exactly where your confidence ends.",[194,315],{},[40,317,319],{"id":318},"what-happens-when-stable-is-broken","What Happens When stable is broken",[56,321,323],{"className":322},[59,60,61,62],[11,324,325,329],{},[66,326],{"alt":327,"src":328},"backups","/continuous-integration/backups.png",[21,330,331],{},"Backups(monkeyuser.com).",[11,333,334],{},"It will happen. An image that passes staging will break in a customer environment for a reason that staging did not surface - a schema migration that assumed a clean database, a dependency that behaves differently on older hardware, a configuration value that was present in staging and absent in the field.",[11,336,337,338,340,341,344,345,347],{},"The recovery flow is: fix on ",[237,339,269],{},", watch it pass staging, cut a new release. ",[237,342,343],{},"v3.0.41"," overwrites ",[237,346,247],{},". Watchtower picks it up on the next poll interval. The customer, who may or may not have noticed anything, is now running the fixed image.",[11,349,350],{},"The window between the broken image landing and the fix arriving is real. Depending on how fast the hotfix moves through staging and how long the Watchtower poll interval is, a customer could be running broken software for anywhere from minutes to hours. There is no remote kill switch. There is no way to reach in and restart a service. There is DWService if the situation is bad enough to warrant it, but that is a support escalation, not a deployment tool.",[11,352,353],{},"This is the honest cost of not controlling the deployment target. You accept a recovery latency that you cannot compress below a certain floor. The mitigation is not a cleverer pipeline. It is investing deeply in staging fidelity and in making sure the image fails loudly rather than silently - health checks that surface problems immediately, startup validation that refuses to run on bad configuration rather than running badly.",[11,355,356],{},"A system that fails loudly is a system that can be fixed. A system that degrades quietly is a system that erodes trust before anyone knows there is a problem.",[194,358],{},[40,360,362],{"id":361},"enterprise-and-standard-different-cadences-same-pattern","Enterprise and standard: different cadences, same pattern",[11,364,365],{},[21,366,367],{},"How do we manage customers that diverge from the main product line with an enterprise license?",[11,369,370,371,374,375,378],{},"NavEngine has two license tiers. Enterprise customers are on a separate release cadence from standard customers. The mechanism is straightforward: separate floating tags on the production registry. ",[237,372,373],{},"core-enterprise:stable"," and ",[237,376,377],{},"core-standard:stable",". The Butane config shipped to each customer points at the appropriate tag. Enterprise releases can go out on a different schedule, carry different feature sets, and move more cautiously than standard releases.",[11,380,381],{},"What prevents a standard customer from pointing Watchtower at the enterprise tag? Mostly friction. The Compose file is baked into the Butane config at provisioning time. There is no SSH access to change it. A customer would need console access and the motivation to go looking - unlikely for most, impossible to rule out for all.",[11,383,384],{},"The proper answer is registry-level access control: pull tokens scoped to the tags each customer is entitled to, issued at license activation and revoked at expiry. This means the registry enforces entitlement, not just the application. An expired license means an expired pull token means no updates, enforced at the point of delivery rather than after the fact.",[11,386,387],{},"This is on the roadmap. For v4, the answer is friction and trust.",[194,389],{},[212,391,392],{},[11,393,394,220,396],{},[217,395,219],{},[21,397,398],{},"License enforcement in self-hosted software is a negotiation between what you can technically control and what you have to trust. You cannot fully control what runs on a machine you do not own. At some point, a sufficiently motivated customer can circumvent almost any enforcement mechanism you build. The goal is not to make circumvention impossible. It is to make compliance easier than circumvention, and to make the value proposition strong enough that the question rarely comes up.",[194,400],{},[40,402,404],{"id":403},"the-license-server-is-not-in-the-update-path","The license server is not in the update path",[11,406,407],{},"One decision I am glad we made early: the licensing server and the Docker registry are separate infrastructure. They do not share a failure domain.",[11,409,410],{},"Watchtower polls the registry. The license server is called from within one of the running containers as part of normal application operation. If the registry is unreachable, the software keeps running. If the license server is unreachable, the backend falls back to its last known state - persisted to disk, not held in memory, so it survives a container restart. The check runs periodically. The grace window is generous enough that a license server outage does not immediately affect customers, but not so generous that expired licenses can run indefinitely.",[11,412,413],{},"This matters because the failure modes compound. A product update that requires a license check to proceed has just made the license server a dependency of your deployment pipeline. Any outage that hits your license infrastructure also hits your ability to ship updates to paying customers. Keeping these paths separate means they fail independently, and independent failures are recoverable in ways that cascading ones are not.",[194,415],{},[11,417,418],{},[217,419,420],{},"What the pipeline actually looks like",[56,422,424],{"className":423},[59,60,61,62],[11,425,426,430],{},[66,427],{"alt":428,"src":429},"continuous","/continuous-integration/draw-continuous-integration.png",[21,431,432,433,436],{},"CI/CD flow diagram (",[217,434,435],{},"drawio",").",[11,438,439,440,442,443,446,447,449],{},"The immutable version tags are not just for auditing. They are the rollback reference. If ",[237,441,283],{}," breaks everything, ",[237,444,445],{},"v3.0.39"," still exists in the registry. I can retag it as ",[237,448,247],{}," manually and customers will roll back on the next poll. This has not been needed yet. It is there for the day it is.",[194,451],{},[212,453,454],{},[11,455,456,220,458],{},[217,457,219],{},[21,459,460],{},"Most CI/CD writing treats deployment as the end of the story. Ship it, watch the metrics, move on. Self-hosted software inverts this. Deployment is the beginning of a period during which software you cannot reach is running in an environment you cannot see, on behalf of a customer whose experience you will only hear about if something goes wrong. The pipeline is not a delivery mechanism. It is a trust mechanism. Every decision in it is a decision about how much you trust the image before it leaves your hands.",[194,462],{},[40,464,466],{"id":465},"three-weeks-out","Three Weeks Out",[11,468,469],{},"NavEngine v4 is three weeks away. The pipeline is running. Staging has held. The tags are in place.",[11,471,472,473],{},"None of that answers the only question that matters: ",[21,474,475],{},"what happens when the software leaves you?",[11,477,478],{},"It will run on machines you have never touched, against data you have never seen, in environments that do not care about your assumptions. By the time it fails, if it fails, it will already be someone else’s problem - and still entirely yours. The customer notices before you do. That is the thing. I am shit scared.",[11,480,481,482,487],{},"I suppose that this is the essence of ",[48,483,486],{"href":484,"rel":485},"https://www.marvinkweyu.net/indulge/why-distributed-systems-field-notes",[52],"these notes"," - to document real systems in real time. This is the inversion self-hosted systems force on you: treating deployment not as the end of control but as the beginning of its absence. ",[11,489,490],{},"So you design for that absence.",[11,492,493],{},"You design for recovery over prevention.",[11,495,496],{},"For visibility over certainty and trust over control.",[11,498,499],{},"Everything else is just what it takes to make that possible.",{"title":136,"searchDepth":137,"depth":137,"links":501},[502,503,504,505,506,507,508],{"id":198,"depth":140,"text":199},{"id":228,"depth":140,"text":229},{"id":259,"depth":140,"text":260},{"id":318,"depth":140,"text":319},{"id":361,"depth":140,"text":362},{"id":403,"depth":140,"text":404},{"id":465,"depth":140,"text":466},"continuous-integration/software_architecture.jpg","2026-04-06","Three weeks from shipping NavEngine v4. No SSH access. No deployment target. Here's how updates reach machines behind a customer's firewall.",{},"/indulge/continuous-integration-for-licensed-software","---\ntitle: CI/CD for Licensed Software You Don't Host\ncoverImage: continuous-integration/software_architecture.jpg\ndate: 2026-04-06\ndescription: Three weeks from shipping NavEngine v4. No SSH access. No deployment target. Here's how updates reach machines behind a customer's firewall.\ndraft: false\nfeatured: false\ngroup: Distributed Systems Field Notes\ninitial: false\nreadTime: 8\nseo:\n  title: CI/CD for Licensed Software You Don't Host\n  description: How to continuously deliver updates to on-premise software you can't SSH into - floating Docker tags, Watchtower, dual-registry pipelines, and license-gated updates from the field.\nseries: true\nseriesCover: false\nsubtitle: Floating tags, Watchtower and the trust contract hidden inside every update\ntags:\n  - distributed systems\n  - software licensing\n  - devops\n  - self-hosted\n  - docker\n---\n\nWe are three weeks away from shipping [NavEngine v4](https://navengine.encha.cloud/), an echo from the previous piece on [Business Driven development](https://www.marvinkweyu.net/indulge/business-minded-development).\n\nI say \"_shipping_\" loosely. There is no deployment script I run, no SSH session I open and no Kubernetes rollout I watch. The software lives on infrastructure I have never seen, behind firewalls I cannot reach, on machines whose specs I do not know. Shipping, in this context, means pushing an image to a registry and trusting that a process running inside a container on a customer's server will eventually notice and do something about it.\n\nThat gap - between what I push and what the customer runs - is what this piece is about.\n\n---\n\n### The Assumption Collapse\n\nEvery CI/CD tool I have ever used was built on a premise so foundational that nobody thought to state it: you control the deployment target. You own the server. You have the keys. Deployment, in the conventional sense, is just automation wrapped around access you already have.\n\nNavEngine is quite the opposite. It exists as a custom image - qcow2 - shipped onto the customer's infrastructure. The customer owns the machine. I do not have SSH access unless I go through DWService and even then, that is a support channel, not a deployment one. Yet somehow, we need to continuously deliver updates to machines we cannot reach, across connections we cannot guarantee, without breaking software that is actively in use.\n\nSo the question became: if you cannot push, how do you deliver?\n\n---\n\n> **Indulge** _The moment you decide to ship software you don't host, you have made a decision with consequences that will follow you for the life of the product. Not just operationally. Architecturally. Every assumption your codebase makes about the environment it runs in now belongs to someone else's infrastructure. That is not a deployment problem. It is a design problem that shows up at deployment time._\n\n---\n\n### The Answer is Pull\n\nWatchtower. It runs as a container alongside the rest of the stack, polls the Docker registry on a configured interval, and when it detects a new image digest on the tag it is watching, it pulls and restarts the relevant containers. No webhook, no push, no SSH. The installation phones home for updates and takes what it finds.\n\nThe key design decision here was the floating tag. Every customer's Butane config ships with `core:stable`. Not `core:v3.0.39`. Not a digest pin. `stable`. When Watchtower polls the registry and sees that `stable` now resolves to a different digest than what is currently running, it pulls. What \"stable\" points to is entirely under my control, from the registry side, without touching anything on the customer's machine.\n\nThis sounds obvious once you say it. It took longer than I would like to admit to get there.\n\n---\n\n### Two Registries, Two floating tags, One gate\n\nHere is the full pipeline as it actually runs.\n\nEvery push to `main` triggers a build on the dev registry. The image gets tagged with a version identifier and a floating tag - `staging`. A staging environment - running the same Compose stack, same Butane configuration, same structure as a customer installation - pulls from `staging`. This is where the image lives until I am satisfied it works.\n\nWhen staging looks good, I create a release. The production registry builds from that release, tags the image with the version (`v3.0.40`) and overwrites the floating `stable` tag. Customer installations, on their next Watchtower poll interval, see a new digest on `stable` and update.\n\nThe critical detail: `stable` is never overwritten by a push to `main`. Only by a release. The staging gate is the only thing standing between a broken image and a customer's running installation. There is no automated rollout percentage, no canary fleet, no gradual traffic shift. The gate is a human decision, made after watching staging run and deciding it is ready.\n\nFor a solo-operated product at this stage, that is the right call. Complexity in release infrastructure that you do not need is just surface area for things to go wrong.\n\n---\n\n> **Indulge** _A staging environment that does not accurately reflect production is a very expensive placebo. It gives you confidence without giving you information. The hardest thing about shipping self-hosted software is that your staging environment runs on infrastructure you understand, with data you created, on a network you control. Your customer's environment is none of those things. No pipeline fully closes that gap. The best you can do is know exactly where your confidence ends._\n\n---\n\n### What Happens When stable is broken\n\n::div{.max-w-2xl.mx-auto.text-center.text-xs}\n![backups](/continuous-integration/backups.png)_Backups(monkeyuser.com)._\n::\n\nIt will happen. An image that passes staging will break in a customer environment for a reason that staging did not surface - a schema migration that assumed a clean database, a dependency that behaves differently on older hardware, a configuration value that was present in staging and absent in the field.\n\nThe recovery flow is: fix on `main`, watch it pass staging, cut a new release. `v3.0.41` overwrites `stable`. Watchtower picks it up on the next poll interval. The customer, who may or may not have noticed anything, is now running the fixed image.\n\nThe window between the broken image landing and the fix arriving is real. Depending on how fast the hotfix moves through staging and how long the Watchtower poll interval is, a customer could be running broken software for anywhere from minutes to hours. There is no remote kill switch. There is no way to reach in and restart a service. There is DWService if the situation is bad enough to warrant it, but that is a support escalation, not a deployment tool.\n\nThis is the honest cost of not controlling the deployment target. You accept a recovery latency that you cannot compress below a certain floor. The mitigation is not a cleverer pipeline. It is investing deeply in staging fidelity and in making sure the image fails loudly rather than silently - health checks that surface problems immediately, startup validation that refuses to run on bad configuration rather than running badly.\n\nA system that fails loudly is a system that can be fixed. A system that degrades quietly is a system that erodes trust before anyone knows there is a problem.\n\n---\n\n### Enterprise and standard: different cadences, same pattern\n\n_How do we manage customers that diverge from the main product line with an enterprise license?_\n\nNavEngine has two license tiers. Enterprise customers are on a separate release cadence from standard customers. The mechanism is straightforward: separate floating tags on the production registry. `core-enterprise:stable` and `core-standard:stable`. The Butane config shipped to each customer points at the appropriate tag. Enterprise releases can go out on a different schedule, carry different feature sets, and move more cautiously than standard releases.\n\nWhat prevents a standard customer from pointing Watchtower at the enterprise tag? Mostly friction. The Compose file is baked into the Butane config at provisioning time. There is no SSH access to change it. A customer would need console access and the motivation to go looking - unlikely for most, impossible to rule out for all.\n\nThe proper answer is registry-level access control: pull tokens scoped to the tags each customer is entitled to, issued at license activation and revoked at expiry. This means the registry enforces entitlement, not just the application. An expired license means an expired pull token means no updates, enforced at the point of delivery rather than after the fact.\n\nThis is on the roadmap. For v4, the answer is friction and trust.\n\n---\n\n> **Indulge** _License enforcement in self-hosted software is a negotiation between what you can technically control and what you have to trust. You cannot fully control what runs on a machine you do not own. At some point, a sufficiently motivated customer can circumvent almost any enforcement mechanism you build. The goal is not to make circumvention impossible. It is to make compliance easier than circumvention, and to make the value proposition strong enough that the question rarely comes up._\n\n---\n\n### The license server is not in the update path\n\nOne decision I am glad we made early: the licensing server and the Docker registry are separate infrastructure. They do not share a failure domain.\n\nWatchtower polls the registry. The license server is called from within one of the running containers as part of normal application operation. If the registry is unreachable, the software keeps running. If the license server is unreachable, the backend falls back to its last known state - persisted to disk, not held in memory, so it survives a container restart. The check runs periodically. The grace window is generous enough that a license server outage does not immediately affect customers, but not so generous that expired licenses can run indefinitely.\n\nThis matters because the failure modes compound. A product update that requires a license check to proceed has just made the license server a dependency of your deployment pipeline. Any outage that hits your license infrastructure also hits your ability to ship updates to paying customers. Keeping these paths separate means they fail independently, and independent failures are recoverable in ways that cascading ones are not.\n\n---\n\n**What the pipeline actually looks like**\n\n::div{.max-w-2xl.mx-auto.text-center.text-xs}\n![continuous](/continuous-integration/draw-continuous-integration.png)_CI/CD flow diagram (**drawio**)._\n::\n\nThe immutable version tags are not just for auditing. They are the rollback reference. If `v3.0.40` breaks everything, `v3.0.39` still exists in the registry. I can retag it as `stable` manually and customers will roll back on the next poll. This has not been needed yet. It is there for the day it is.\n\n---\n\n> **Indulge** _Most CI/CD writing treats deployment as the end of the story. Ship it, watch the metrics, move on. Self-hosted software inverts this. Deployment is the beginning of a period during which software you cannot reach is running in an environment you cannot see, on behalf of a customer whose experience you will only hear about if something goes wrong. The pipeline is not a delivery mechanism. It is a trust mechanism. Every decision in it is a decision about how much you trust the image before it leaves your hands._\n\n---\n\n### Three Weeks Out\n\nNavEngine v4 is three weeks away. The pipeline is running. Staging has held. The tags are in place.\n\nNone of that answers the only question that matters: _what happens when the software leaves you?_\n\nIt will run on machines you have never touched, against data you have never seen, in environments that do not care about your assumptions. By the time it fails, if it fails, it will already be someone else’s problem - and still entirely yours. The customer notices before you do. That is the thing. I am shit scared.\n\nI suppose that this is the essence of [these notes](https://www.marvinkweyu.net/indulge/why-distributed-systems-field-notes) - to document real systems in real time. This is the inversion self-hosted systems force on you: treating deployment not as the end of control but as the beginning of its absence. \n\nSo you design for that absence.\n\nYou design for recovery over prevention.\n\nFor visibility over certainty and trust over control.\n\nEverything else is just what it takes to make that possible.\n",{"title":165,"description":516},"How to continuously deliver updates to on-premise software you can't SSH into - floating Docker tags, Watchtower, dual-registry pipelines, and license-gated updates from the field.","indulge/continuous-integration-for-licensed-software","Floating tags, Watchtower and the trust contract hidden inside every update",[159,520,521,522,523],"software licensing","devops","self-hosted","docker","meHkqo48tYpQaHMbPRSdebg3p8dq7UDNy2pk1Nz5C-M",{"id":526,"title":148,"body":527,"coverImage":152,"date":531,"description":532,"draft":146,"extension":147,"featured":146,"group":148,"initial":146,"meta":533,"navigation":150,"path":534,"rank":152,"rawbody":535,"readTime":536,"seo":537,"series":150,"seriesCover":150,"seriesOrder":152,"seriesSlug":152,"slug":152,"stem":538,"subtitle":539,"tags":540,"__hash__":541},"indulge/indulge/distributed_systems_field_notes.md",{"type":8,"value":528,"toc":529},[],{"title":136,"searchDepth":137,"depth":137,"links":530},[],"2026-06-10T10:22:51.676Z","Hard-won insights from architecting and operating systems at scale. Patterns, anti-patterns and war stories.",{},"/indulge/distributed_systems_field_notes","---\ntitle: Distributed Systems Field Notes\nseries: true\nseriesCover: true\ngroup: Distributed Systems Field Notes\ndescription: Hard-won insights from architecting and operating systems at scale. Patterns, anti-patterns and war stories.\nfeatured: false\ndraft: false\nsubtitle: Scalability\ntags:\n  - distributed systems\n---\n",5,{"title":148,"description":532},"indulge/distributed_systems_field_notes","Scalability",[159],"q8leXl8e6PwSYhPCWGY9ghOXxWLC95qZbxhP081d23c",{"id":543,"title":544,"body":545,"coverImage":152,"date":549,"description":550,"draft":146,"extension":147,"featured":146,"group":551,"initial":146,"meta":552,"navigation":150,"path":553,"rank":152,"rawbody":554,"readTime":536,"seo":555,"series":150,"seriesCover":150,"seriesOrder":152,"seriesSlug":152,"slug":152,"stem":556,"subtitle":557,"tags":558,"__hash__":563},"indulge/indulge/homelabbing.md","Self-Hosting & Homelabs",{"type":8,"value":546,"toc":547},[],{"title":136,"searchDepth":137,"depth":137,"links":548},[],"31/10/2025","Practical notes on self-hosting and homelabbing - running production-grade services on personal infrastructure, and learning what the cloud usually hides.","Infrastructure",{},"/indulge/homelabbing","---\ntitle: Self-Hosting & Homelabs\nseries: true\nseriesCover: true\ngroup: Infrastructure\ndescription: Practical notes on self-hosting and homelabbing - running production-grade services on personal infrastructure, and learning what the cloud usually hides.\nfeatured: false\ndraft: false\ndate: 31/10/2025\nsubtitle: Running production - grade systems on personal infrastructure\ntags:\n  - self-hosting\n  - homelab\n  - infrastructure\n  - devops\n  - containerization\n---\n",{"title":544,"description":550},"indulge/homelabbing","Running production - grade systems on personal infrastructure",[559,560,561,521,562],"self-hosting","homelab","infrastructure","containerization","e5fgSMrF9oGV2uUQ5Y-FW7CDso8nAH93dOVB0vbNnhU",{"id":565,"title":566,"body":567,"coverImage":152,"date":1348,"description":1349,"draft":146,"extension":147,"featured":146,"group":551,"initial":146,"meta":1350,"navigation":150,"path":1351,"rank":152,"rawbody":1352,"readTime":536,"seo":1353,"series":150,"seriesCover":146,"seriesOrder":152,"seriesSlug":152,"slug":152,"stem":1354,"subtitle":1355,"tags":1356,"__hash__":1358},"indulge/indulge/keeping-the-homelab-alive.md","Keeping the homelab Alive",{"type":8,"value":568,"toc":1344},[569,572,575,578,581,584,587,593,596,599,603,610,617,637,647,704,719,741,748,761,770,773,810,815,818,831,839,845,848,872,876,879,1296,1306,1330,1333,1337,1340],[11,570,571],{},"One of the challenges that comes from getting yourself an enterprise minicomputer is the configuration set by the selling company. For my case, it was the BIOS lock - a toggle that inhibited my ability to have continuous uptime.",[11,573,574],{},"For this specific series, I had the following options to reset this:",[11,576,577],{},"1. Unplug and plug back in the CMOS battery, effectively resetting the BIOS lock.",[11,579,580],{},"2. Removing the PCMOS plug, holding the reset option and placing it back in.",[11,582,583],{},"3. Flushing the BIOS and installing a new one.",[11,585,586],{},"With option 1 and 2 not working for my particular use case and option 3 being out of my reach at that said time, I chose a fourth option. The workaround that came to mind was using wake on LAN. I figured, even if I lost power and it came back up, I should be able to power up the computer using another device on the same network.",[40,588,590],{"id":589},"considerations",[217,591,592],{},"Considerations",[11,594,595],{},"1. What happens should I restart my homelab? Would it remember my configuration to wake up on LAN on command?",[11,597,598],{},"2. How would I connect to it within my home network?",[40,600,602],{"id":601},"approach","Approach",[604,605,607],"h4",{"id":606},"create-a-daemon-to-autostart-wake-on-lan-for-the-homelab",[217,608,609],{},"Create a daemon to autostart wake on LAN for the homelab",[11,611,612,613,616],{},"I connected my setup to my router via ethernet as I would always be assured that that was on. Figuring the interface with which it connected to the home network was as simple as using ",[237,614,615],{},"ip",":",[618,619,623],"pre",{"className":620,"code":621,"language":622,"meta":136,"style":136},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","ip link\n","bash",[237,624,625],{"__ignoreMap":136},[626,627,630,633],"span",{"class":628,"line":629},"line",1,[626,631,615],{"class":632},"sBMFI",[626,634,636],{"class":635},"sfazB"," link\n",[11,638,639,640,643,644,182],{},"From this, I would then pick the connection with ",[237,641,642],{},"\u003CBROADCAST,MULTICAST>"," to check its support. In my case, ",[237,645,646],{},"eno1",[618,648,650],{"className":620,"code":649,"language":622,"meta":136,"style":136},"sudo ethtool eno1\n\n# output contained\n\nSupports Wake-on: pumbg\n\nWake-on: g\n",[237,651,652,663,668,674,679,690,695],{"__ignoreMap":136},[626,653,654,657,660],{"class":628,"line":629},[626,655,656],{"class":632},"sudo",[626,658,659],{"class":635}," ethtool",[626,661,662],{"class":635}," eno1\n",[626,664,665],{"class":628,"line":137},[626,666,667],{"emptyLinePlaceholder":150},"\n",[626,669,670],{"class":628,"line":140},[626,671,673],{"class":672},"sHwdD","# output contained\n",[626,675,677],{"class":628,"line":676},4,[626,678,667],{"emptyLinePlaceholder":150},[626,680,681,684,687],{"class":628,"line":536},[626,682,683],{"class":632},"Supports",[626,685,686],{"class":635}," Wake-on:",[626,688,689],{"class":635}," pumbg\n",[626,691,693],{"class":628,"line":692},6,[626,694,667],{"emptyLinePlaceholder":150},[626,696,698,701],{"class":628,"line":697},7,[626,699,700],{"class":632},"Wake-on:",[626,702,703],{"class":635}," g\n",[11,705,706,707,710,711,714,715,718],{},"Great, wake on LAN is supported ",[237,708,709],{},"wake-on: g",") and it can support multiple triggers",[237,712,713],{},"Wake-on: pumbg","). Should it have been ",[237,716,717],{},"Wake-on: d",") for not supported, I would have enabled it with:",[618,720,722],{"className":620,"code":721,"language":622,"meta":136,"style":136},"sudo ethtool -s eno1 wol g\n",[237,723,724],{"__ignoreMap":136},[626,725,726,728,730,733,736,739],{"class":628,"line":629},[626,727,656],{"class":632},[626,729,659],{"class":635},[626,731,732],{"class":635}," -s",[626,734,735],{"class":635}," eno1",[626,737,738],{"class":635}," wol",[626,740,703],{"class":635},[11,742,743,744,747],{},"Because the NIC of the system will be reset to ",[237,745,746],{},"d"," - disabled) on each restart, I needed to enable this on reboot. I created an internal service for this as:",[618,749,751],{"className":620,"code":750,"language":622,"meta":136,"style":136},"nano /etc/systemd/system/wol.service\n",[237,752,753],{"__ignoreMap":136},[626,754,755,758],{"class":628,"line":629},[626,756,757],{"class":632},"nano",[626,759,760],{"class":635}," /etc/systemd/system/wol.service\n",[618,762,768],{"className":763,"code":765,"filename":766,"language":767,"meta":136},[764],"language-text","[Unit]\n\nDescription=Enable Wake-on-LAN\nAfter=network.target\n\n[Service]\nType=oneshot\nExecStart=/sbin/ethtool -s eno1 wol g\n\n[Install]\nWantedBy=multi-user.target\n","wol.service","text",[237,769,765],{"__ignoreMap":136},[11,771,772],{},"Then I recognize and enable it with:",[618,774,776],{"className":620,"code":775,"language":622,"meta":136,"style":136},"sudo systemctl daemon-reload\nsudo systemctl enable wol.service\nsystemct start wol.service\n",[237,777,778,788,800],{"__ignoreMap":136},[626,779,780,782,785],{"class":628,"line":629},[626,781,656],{"class":632},[626,783,784],{"class":635}," systemctl",[626,786,787],{"class":635}," daemon-reload\n",[626,789,790,792,794,797],{"class":628,"line":137},[626,791,656],{"class":632},[626,793,784],{"class":635},[626,795,796],{"class":635}," enable",[626,798,799],{"class":635}," wol.service\n",[626,801,802,805,808],{"class":628,"line":140},[626,803,804],{"class":632},"systemct",[626,806,807],{"class":635}," start",[626,809,799],{"class":635},[11,811,812],{},[217,813,814],{},"Pinging the homelab from other devices in the same home network:",[11,816,817],{},"Since the system is already on the home network with ethernet and always plugged in, I can call wake on lan using its MAC address. I would just need to install wake on LAN on my laptop or other device. On Arch:",[618,819,821],{"className":620,"code":820,"filename":622,"language":622,"meta":136,"style":136},"yay wakeonlan\n",[237,822,823],{"__ignoreMap":136},[626,824,825,828],{"class":628,"line":629},[626,826,827],{"class":632},"yay",[626,829,830],{"class":635}," wakeonlan\n",[11,832,833,834,182],{},"On my android devices, I chose to install a ready made solution. I was not going to start building again ",[48,835,838],{"href":836,"rel":837},"https://play.google.com/store/apps/details?id=co.uk.mrwebb.wakeonlan",[52],"Wake On Lan",[11,840,841,842,844],{},"I then retrieved the MAC address using the same ",[237,843,615],{}," command from earlier.",[11,846,847],{},"Now, waking my computer from a power outage should work as simple as:",[618,849,851],{"className":620,"code":850,"language":622,"meta":136,"style":136},"wakeonlan \u003Cmac-address>\n",[237,852,853],{"__ignoreMap":136},[626,854,855,858,862,865,869],{"class":628,"line":629},[626,856,857],{"class":632},"wakeonlan",[626,859,861],{"class":860},"sMK4o"," \u003C",[626,863,864],{"class":635},"mac-addres",[626,866,868],{"class":867},"sTEyZ","s",[626,870,871],{"class":860},">\n",[604,873,875],{"id":874},"a-step-further","A step further",[11,877,878],{},"I took it a step further and updated my aliases. In this case, creating a bash script that would ping my home server when called and only stop after 30 minutes of failure or when it got its first successful response. For reference:",[618,880,883],{"className":620,"code":881,"filename":882,"language":622,"meta":136,"style":136},"#!/bin/bash\n\n# wake-on-lan-script\nMAC_ADDRESS=\"mac-address\"\nHOSTNAME=\"homelab-hostname\"\nSERVER_IP=\"homelab-ip\"\n\necho \"Waking up $HOSTNAME ($MAC_ADDRESS)...\"\nwakeonlan \"$MAC_ADDRESS\"\n\nsleep 45\necho \"Checking if $HOSTNAME is up (will timeout after 30 minutes)...\"\n\nSTART_TIME=$(date +%s)\n\nTIMEOUT=$((30 * 60))  # 30 minutes in seconds\n\nwhile true; do\n   if ping -c 1 -W 1 \"$SERVER_IP\" &> /dev/null; then\n       echo \"$HOSTNAME is now online.\"\n       break\n   fi\n\n   NOW=$(date +%s)\n   ELAPSED=$((NOW - START_TIME))\n\n   if [ \"$ELAPSED\" -ge \"$TIMEOUT\" ]; then\n       echo \"Timeout reached. $HOSTNAME did not come online within 30 minutes.\"\n       break\n   fi\n   sleep 5\n\ndone\n","wake-on-lan-script",[237,884,885,890,894,899,916,930,944,948,974,985,990,1000,1017,1022,1040,1045,1081,1086,1102,1140,1155,1161,1167,1172,1189,1212,1217,1249,1266,1271,1276,1285,1290],{"__ignoreMap":136},[626,886,887],{"class":628,"line":629},[626,888,889],{"class":672},"#!/bin/bash\n",[626,891,892],{"class":628,"line":137},[626,893,667],{"emptyLinePlaceholder":150},[626,895,896],{"class":628,"line":140},[626,897,898],{"class":672},"# wake-on-lan-script\n",[626,900,901,904,907,910,913],{"class":628,"line":676},[626,902,903],{"class":867},"MAC_ADDRESS",[626,905,906],{"class":860},"=",[626,908,909],{"class":860},"\"",[626,911,912],{"class":635},"mac-address",[626,914,915],{"class":860},"\"\n",[626,917,918,921,923,925,928],{"class":628,"line":536},[626,919,920],{"class":867},"HOSTNAME",[626,922,906],{"class":860},[626,924,909],{"class":860},[626,926,927],{"class":635},"homelab-hostname",[626,929,915],{"class":860},[626,931,932,935,937,939,942],{"class":628,"line":692},[626,933,934],{"class":867},"SERVER_IP",[626,936,906],{"class":860},[626,938,909],{"class":860},[626,940,941],{"class":635},"homelab-ip",[626,943,915],{"class":860},[626,945,946],{"class":628,"line":697},[626,947,667],{"emptyLinePlaceholder":150},[626,949,950,954,957,960,963,966,969,972],{"class":628,"line":154},[626,951,953],{"class":952},"s2Zo4","echo",[626,955,956],{"class":860}," \"",[626,958,959],{"class":635},"Waking up ",[626,961,962],{"class":867},"$HOSTNAME",[626,964,965],{"class":635}," (",[626,967,968],{"class":867},"$MAC_ADDRESS",[626,970,971],{"class":635},")...",[626,973,915],{"class":860},[626,975,977,979,981,983],{"class":628,"line":976},9,[626,978,857],{"class":632},[626,980,956],{"class":860},[626,982,968],{"class":867},[626,984,915],{"class":860},[626,986,988],{"class":628,"line":987},10,[626,989,667],{"emptyLinePlaceholder":150},[626,991,993,996],{"class":628,"line":992},11,[626,994,995],{"class":632},"sleep",[626,997,999],{"class":998},"sbssI"," 45\n",[626,1001,1003,1005,1007,1010,1012,1015],{"class":628,"line":1002},12,[626,1004,953],{"class":952},[626,1006,956],{"class":860},[626,1008,1009],{"class":635},"Checking if ",[626,1011,962],{"class":867},[626,1013,1014],{"class":635}," is up (will timeout after 30 minutes)...",[626,1016,915],{"class":860},[626,1018,1020],{"class":628,"line":1019},13,[626,1021,667],{"emptyLinePlaceholder":150},[626,1023,1025,1028,1031,1034,1037],{"class":628,"line":1024},14,[626,1026,1027],{"class":867},"START_TIME",[626,1029,1030],{"class":860},"=$(",[626,1032,1033],{"class":632},"date",[626,1035,1036],{"class":635}," +%s",[626,1038,1039],{"class":860},")\n",[626,1041,1043],{"class":628,"line":1042},15,[626,1044,667],{"emptyLinePlaceholder":150},[626,1046,1048,1051,1054,1057,1060,1063,1066,1069,1072,1075,1078],{"class":628,"line":1047},16,[626,1049,1050],{"class":867},"TIMEOUT",[626,1052,1053],{"class":860},"=$((",[626,1055,1056],{"class":632},"30",[626,1058,1059],{"class":867}," *",[626,1061,1062],{"class":998}," 60",[626,1064,1065],{"class":860},"))",[626,1067,1068],{"class":632},"  #",[626,1070,1071],{"class":998}," 30",[626,1073,1074],{"class":635}," minutes",[626,1076,1077],{"class":635}," in",[626,1079,1080],{"class":635}," seconds\n",[626,1082,1084],{"class":628,"line":1083},17,[626,1085,667],{"emptyLinePlaceholder":150},[626,1087,1089,1093,1096,1099],{"class":628,"line":1088},18,[626,1090,1092],{"class":1091},"s7zQu","while",[626,1094,1095],{"class":952}," true",[626,1097,1098],{"class":860},";",[626,1100,1101],{"class":1091}," do\n",[626,1103,1105,1108,1111,1114,1117,1120,1122,1124,1127,1129,1132,1135,1137],{"class":628,"line":1104},19,[626,1106,1107],{"class":632},"   if",[626,1109,1110],{"class":635}," ping",[626,1112,1113],{"class":635}," -c",[626,1115,1116],{"class":998}," 1",[626,1118,1119],{"class":635}," -W",[626,1121,1116],{"class":998},[626,1123,956],{"class":860},[626,1125,1126],{"class":867},"$SERVER_IP",[626,1128,909],{"class":860},[626,1130,1131],{"class":860}," &>",[626,1133,1134],{"class":867}," /dev/null",[626,1136,1098],{"class":860},[626,1138,1139],{"class":1091}," then\n",[626,1141,1143,1146,1148,1150,1153],{"class":628,"line":1142},20,[626,1144,1145],{"class":632},"       echo",[626,1147,956],{"class":860},[626,1149,962],{"class":867},[626,1151,1152],{"class":635}," is now online.",[626,1154,915],{"class":860},[626,1156,1158],{"class":628,"line":1157},21,[626,1159,1160],{"class":632},"       break\n",[626,1162,1164],{"class":628,"line":1163},22,[626,1165,1166],{"class":632},"   fi\n",[626,1168,1170],{"class":628,"line":1169},23,[626,1171,667],{"emptyLinePlaceholder":150},[626,1173,1175,1178,1180,1183,1185,1187],{"class":628,"line":1174},24,[626,1176,1177],{"class":632},"   NOW",[626,1179,906],{"class":635},[626,1181,1182],{"class":867},"$(",[626,1184,1033],{"class":635},[626,1186,1036],{"class":635},[626,1188,1039],{"class":867},[626,1190,1192,1195,1197,1200,1203,1206,1209],{"class":628,"line":1191},25,[626,1193,1194],{"class":632},"   ELAPSED",[626,1196,906],{"class":635},[626,1198,1199],{"class":867},"$((",[626,1201,1202],{"class":635},"NOW",[626,1204,1205],{"class":635}," -",[626,1207,1208],{"class":635}," START_TIME",[626,1210,1211],{"class":867},"))\n",[626,1213,1215],{"class":628,"line":1214},26,[626,1216,667],{"emptyLinePlaceholder":150},[626,1218,1220,1222,1225,1227,1230,1232,1235,1237,1240,1242,1245,1247],{"class":628,"line":1219},27,[626,1221,1107],{"class":632},[626,1223,1224],{"class":867}," [ ",[626,1226,909],{"class":860},[626,1228,1229],{"class":867},"$ELAPSED",[626,1231,909],{"class":860},[626,1233,1234],{"class":635}," -ge",[626,1236,956],{"class":860},[626,1238,1239],{"class":867},"$TIMEOUT",[626,1241,909],{"class":860},[626,1243,1244],{"class":635}," ]",[626,1246,1098],{"class":860},[626,1248,1139],{"class":1091},[626,1250,1252,1254,1256,1259,1261,1264],{"class":628,"line":1251},28,[626,1253,1145],{"class":632},[626,1255,956],{"class":860},[626,1257,1258],{"class":635},"Timeout reached. ",[626,1260,962],{"class":867},[626,1262,1263],{"class":635}," did not come online within 30 minutes.",[626,1265,915],{"class":860},[626,1267,1269],{"class":628,"line":1268},29,[626,1270,1160],{"class":632},[626,1272,1274],{"class":628,"line":1273},30,[626,1275,1166],{"class":632},[626,1277,1279,1282],{"class":628,"line":1278},31,[626,1280,1281],{"class":632},"   sleep",[626,1283,1284],{"class":998}," 5\n",[626,1286,1288],{"class":628,"line":1287},32,[626,1289,667],{"emptyLinePlaceholder":150},[626,1291,1293],{"class":628,"line":1292},33,[626,1294,1295],{"class":1091},"done\n",[11,1297,1298,1299,1302,1303],{},"I would then update my ",[237,1300,1301],{},"~/.zshrc"," with my alias call. I just called it ",[237,1304,1305],{},"home",[618,1307,1310],{"className":620,"code":1308,"filename":1309,"language":622,"meta":136,"style":136},"alias home=\"path-to-wake-on-lan-script\"\n","zshrc",[237,1311,1312],{"__ignoreMap":136},[626,1313,1314,1318,1321,1323,1325,1328],{"class":628,"line":629},[626,1315,1317],{"class":1316},"spNyl","alias",[626,1319,1320],{"class":867}," home",[626,1322,906],{"class":860},[626,1324,909],{"class":860},[626,1326,1327],{"class":635},"path-to-wake-on-lan-script",[626,1329,915],{"class":860},[11,1331,1332],{},"That was pretty much it.",[604,1334,1336],{"id":1335},"the-future","The future",[11,1338,1339],{},"My current setup works as I expect. I do, however, foresee a scenario where I would like to wake the system when not connected to my home network. I could either use wireguard with a rasberry pi or setup an ESP32 that would always try to ping it when plugged in.",[1341,1342,1343],"style",{},"html pre.shiki code .sBMFI, html code.shiki .sBMFI{--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B}html pre.shiki code .sfazB, html code.shiki .sfazB{--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D}html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sHwdD, html code.shiki .sHwdD{--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic}html pre.shiki code .sMK4o, html code.shiki .sMK4o{--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF}html pre.shiki code .sTEyZ, html code.shiki .sTEyZ{--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8}html pre.shiki code .s2Zo4, html code.shiki .s2Zo4{--shiki-light:#6182B8;--shiki-default:#82AAFF;--shiki-dark:#82AAFF}html pre.shiki code .sbssI, html code.shiki .sbssI{--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C}html pre.shiki code .s7zQu, html code.shiki .s7zQu{--shiki-light:#39ADB5;--shiki-light-font-style:italic;--shiki-default:#89DDFF;--shiki-default-font-style:italic;--shiki-dark:#89DDFF;--shiki-dark-font-style:italic}html pre.shiki code .spNyl, html code.shiki .spNyl{--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA}",{"title":136,"searchDepth":137,"depth":137,"links":1345},[1346,1347],{"id":589,"depth":140,"text":592},{"id":601,"depth":140,"text":602},"2025-01-31","Discover how to maintain uptime in your homelab with a BIOS lock workaround and wake on LAN setup",{},"/indulge/keeping-the-homelab-alive","---\ntitle: Keeping the homelab Alive\nseries: true\ngroup: Infrastructure\nsubtitle: Maintain a Reliable System\ndescription: Discover how to maintain uptime in your homelab with a BIOS lock\n  workaround and wake on LAN setup\ndate: 2025-01-31\ntags: [homelabbing]\n---\n\nOne of the challenges that comes from getting yourself an enterprise minicomputer is the configuration set by the selling company. For my case, it was the BIOS lock - a toggle that inhibited my ability to have continuous uptime.\n\nFor this specific series, I had the following options to reset this:\n\n1\\. Unplug and plug back in the CMOS battery, effectively resetting the BIOS lock.\n\n2\\. Removing the PCMOS plug, holding the reset option and placing it back in.\n\n3\\. Flushing the BIOS and installing a new one.\n\nWith option 1 and 2 not working for my particular use case and option 3 being out of my reach at that said time, I chose a fourth option. The workaround that came to mind was using wake on LAN. I figured, even if I lost power and it came back up, I should be able to power up the computer using another device on the same network.\n\n### **Considerations**\n\n1\\. What happens should I restart my homelab? Would it remember my configuration to wake up on LAN on command?\n\n2\\. How would I connect to it within my home network?\n\n### Approach\n\n#### **Create a daemon to autostart wake on LAN for the homelab**\n\nI connected my setup to my router via ethernet as I would always be assured that that was on. Figuring the interface with which it connected to the home network was as simple as using `ip`:\n\n```bash\nip link\n```\n\nFrom this, I would then pick the connection with `\u003CBROADCAST,MULTICAST>` to check its support. In my case, `eno1`.\n\n```bash\nsudo ethtool eno1\n\n# output contained\n\nSupports Wake-on: pumbg\n\nWake-on: g\n```\n\nGreat, wake on LAN is supported `wake-on: g`) and it can support multiple triggers`Wake-on: pumbg`). Should it have been `Wake-on: d`) for not supported, I would have enabled it with:\n\n```bash\nsudo ethtool -s eno1 wol g\n```\n\nBecause the NIC of the system will be reset to `d` - disabled) on each restart, I needed to enable this on reboot. I created an internal service for this as:\n\n```bash\nnano /etc/systemd/system/wol.service\n```\n\n```text [wol.service]\n[Unit]\n\nDescription=Enable Wake-on-LAN\nAfter=network.target\n\n[Service]\nType=oneshot\nExecStart=/sbin/ethtool -s eno1 wol g\n\n[Install]\nWantedBy=multi-user.target\n```\n\nThen I recognize and enable it with:\n\n```bash\nsudo systemctl daemon-reload\nsudo systemctl enable wol.service\nsystemct start wol.service\n```\n\n**Pinging the homelab from other devices in the same home network:**\n\nSince the system is already on the home network with ethernet and always plugged in, I can call wake on lan using its MAC address. I would just need to install wake on LAN on my laptop or other device. On Arch:\n\n```bash [bash]\nyay wakeonlan\n```\n\nOn my android devices, I chose to install a ready made solution. I was not going to start building again [Wake On Lan](https://play.google.com/store/apps/details?id=co.uk.mrwebb.wakeonlan).\n\nI then retrieved the MAC address using the same `ip` command from earlier.\n\nNow, waking my computer from a power outage should work as simple as:\n\n```bash\nwakeonlan \u003Cmac-address>\n```\n\n#### A step further\n\nI took it a step further and updated my aliases. In this case, creating a bash script that would ping my home server when called and only stop after 30 minutes of failure or when it got its first successful response. For reference:\n\n```bash [wake-on-lan-script]\n#!/bin/bash\n\n# wake-on-lan-script\nMAC_ADDRESS=\"mac-address\"\nHOSTNAME=\"homelab-hostname\"\nSERVER_IP=\"homelab-ip\"\n\necho \"Waking up $HOSTNAME ($MAC_ADDRESS)...\"\nwakeonlan \"$MAC_ADDRESS\"\n\nsleep 45\necho \"Checking if $HOSTNAME is up (will timeout after 30 minutes)...\"\n\nSTART_TIME=$(date +%s)\n\nTIMEOUT=$((30 * 60))  # 30 minutes in seconds\n\nwhile true; do\n   if ping -c 1 -W 1 \"$SERVER_IP\" &> /dev/null; then\n       echo \"$HOSTNAME is now online.\"\n       break\n   fi\n\n   NOW=$(date +%s)\n   ELAPSED=$((NOW - START_TIME))\n\n   if [ \"$ELAPSED\" -ge \"$TIMEOUT\" ]; then\n       echo \"Timeout reached. $HOSTNAME did not come online within 30 minutes.\"\n       break\n   fi\n   sleep 5\n\ndone\n```\n\nI would then update my `~/.zshrc` with my alias call. I just called it `home`\n\n```bash [zshrc]\nalias home=\"path-to-wake-on-lan-script\"\n```\n\nThat was pretty much it.\n\n#### The future\n\nMy current setup works as I expect. I do, however, foresee a scenario where I would like to wake the system when not connected to my home network. I could either use wireguard with a rasberry pi or setup an ESP32 that would always try to ping it when plugged in.\n",{"title":566,"description":1349},"indulge/keeping-the-homelab-alive","Maintain a Reliable System",[1357],"homelabbing","C_AU4EvCAvdFKk_9hGIguD_w1sx__HbA8gEi4htydLo",{"id":1360,"title":1361,"body":1362,"coverImage":1633,"date":1634,"description":1635,"draft":146,"extension":147,"featured":150,"group":148,"initial":146,"meta":1636,"navigation":150,"path":1637,"rank":152,"rawbody":1638,"readTime":697,"seo":1639,"series":150,"seriesCover":146,"seriesOrder":152,"seriesSlug":152,"slug":152,"stem":1640,"subtitle":1544,"tags":1641,"__hash__":1643},"indulge/indulge/one-bucket-three-jobs.md","One Bucket, Three Jobs",{"type":8,"value":1363,"toc":1625},[1364,1373,1376,1379,1382,1387,1390,1411,1414,1417,1421,1424,1431,1434,1437,1448,1455,1464,1468,1471,1487,1490,1493,1496,1506,1510,1513,1520,1523,1526,1529,1532,1535,1538,1541,1545,1548,1551,1554,1564,1567,1570,1573,1576,1578,1592,1601,1603,1607,1610,1613,1616,1619,1622],[11,1365,1366,1367,1372],{},"The ",[48,1368,1371],{"href":1369,"rel":1370},"https://www.marvinkweyu.net/indulge/continuous-integration-for-licensed-software",[52],"last piece"," ended with an image leaving my hands and landing somewhere I cannot see. I described shipping NavEngine as pushing a container image to a registry and trusting that a process on a customer's server will eventually notice.",[11,1374,1375],{},"That is accurate. It is also incomplete.",[11,1377,1378],{},"So far, there are two states for NavEngine; the hosted version - whose database I am responsible for whether it fails at three in the afternoon or three in the morning and the one a customer installs.",[11,1380,1381],{},"For this read, I share the story of what I built to hold it all together and why it ended up being more than a place to put files.",[1383,1384,1386],"h2",{"id":1385},"the-decision-i-didnt-know-i-was-making","The Decision I Didn't Know I Was Making",[11,1388,1389],{},"I had a lot of questions at the start of the project.",[1391,1392,1393,1397,1400,1403],"ul",{},[1394,1395,1396],"li",{},"How would the customers download the product?",[1394,1398,1399],{},"Where would it be stored?",[1394,1401,1402],{},"How do companies like Canonical handle distribution?",[1394,1404,1405,1406,1410],{},"What does ",[48,1407,1409],{"href":179,"rel":1408},[52],"distribution"," actually imply?",[11,1412,1413],{},"I had set out to solve a storage problem. I needed a place that was reliable, versioned, and reachable from anywhere I needed to reach it from. S3 was the obvious answer. Most things are obvious in retrospect. I did not think much about the decision at the time.",[11,1415,1416],{},"What I did not anticipate was that a storage decision is also an architecture decision, and an architecture decision at the distribution layer has a way of spreading.",[1383,1418,1420],{"id":1419},"job-one-the-product-suite","Job One: The Product Suite",[11,1422,1423],{},"NavEngine started as an ISO. Before a customer would run anything on their infrastructure, they needed to have an existing artifact . They needed somewhere they could fetch it from, with integrity guarantees, across whatever network conditions exist on their end.",[212,1425,1426],{},[11,1427,1428,1430],{},[217,1429,219],{},"\nThis raised a different constraint: multiple products would need the same guarantees.",[11,1432,1433],{},"I needed a place to store iterations as I built towards UAT - User Acceptance Testing. The registry handled images. The bucket handled everything else.",[11,1435,1436],{},"The decision to version the artifacts explicitly - not overwrite, not float - came from the same reasoning behind the floating tag in the update pipeline: I need to know exactly what was shipped.",[56,1438,1440],{"className":1439},[59,60,61,62],[11,1441,1442,1445],{},[66,1443],{"alt":68,"src":1444},"/buckets/iteration.jpeg",[21,1446,1447],{},"Versioning, but without guarantees..",[11,1449,1450,1451,1454],{},"If ",[237,1452,1453],{},"v4.5.0"," ever needed to be reproduced or rolled back to, it should already exist. Not rebuilt. Not reconstructed. Present.",[212,1456,1457],{},[11,1458,1459,220,1461],{},[217,1460,219],{},[21,1462,1463],{},"The hardest part of distributing software you do not host is preserving the integrity of what you shipped. Tags drift. References move. The artifact you think you gave a customer can diverge from what they actually received. Versioned, immutable storage is the only reliable paper trail when something breaks on a system you cannot access.",[1383,1465,1467],{"id":1466},"job-two-the-backup","Job Two: The Backup",[11,1469,1470],{},"We initially served backups of NavEngine by running weekly and monthly VM snapshots. It was tedious, human-dependent and error-prone.",[212,1472,1473,1479,1482],{},[11,1474,1475,1478],{},[217,1476,1477],{},"Example:","\nIf an engineer needed to manually update a database record, they would have to manually create a backup for that before that individual operation. This backup would then live on the same machine.",[11,1480,1481],{},"This meant that while the backup was present, if anything were to hit the fan, it was game over.",[11,1483,1484],{},[21,1485,1486],{},"I have been here. The only grace I had was that it was on the staging environment. The story of how I got back is for another day.",[11,1488,1489],{},"The issue wasn’t that we lacked backups. It was that restoring them required the same precision that introduced the risk.",[11,1491,1492],{},"A backup that lives in a different system than your product artifacts is a backup that gets forgotten about during an incident, when you are moving fast and your attention is already split. Having everything in one place means one set of access credentials, one retention policy to reason about, one place to look when something is wrong.",[11,1494,1495],{},"It was anything but elegant. It was reliable.",[212,1497,1498,1503],{},[11,1499,1500,1502],{},[217,1501,219],{},"\nBackup strategies fail at the moment of failure, not at the moment of setup. The question to ask is not \"do I have backups\" but \"can I restore from them at two in the morning, under pressure, with the right people asking questions?\"",[11,1504,1505],{},"If the answer is uncertain, the backup strategy is incomplete regardless of how often the job runs.",[1383,1507,1509],{"id":1508},"job-three-the-agnostic-product-registry","Job Three: The Agnostic Product Registry",[11,1511,1512],{},"This is the job I had not planned for.",[212,1514,1515],{},[11,1516,1517],{},[21,1518,1519],{},"What if we had other products to ship that had the same constraints?",[11,1521,1522],{},"Beyond hosting the versioned images of NavEngine, the storage option needed to be agnostic enough to store other images while retaining the ability to version those products too.",[11,1524,1525],{},"Somewhere in the process of building the first two jobs, I noticed that the structure I was laying down did not have to be NavEngine-specific. The bucket had opinions - a versioning scheme, an artifact layout, a naming convention - but those opinions were not tied to NavEngine. They were tied to the idea of a distributable product. Any product that needed the same things NavEngine needs - versioned artifacts, clean separation between releases, a retrievable history - could sit in the same bucket structure without the structure caring what the product was.",[11,1527,1528],{},"I did not set out to build a product registry. I set out to store some files. The registry emerged from the constraints.",[11,1530,1531],{},"As the team builds BusinessAI, the need to account for portability became obvious. Different pricing. Different licensing. Same distribution problem.",[11,1533,1534],{},"This matters because distribution surfaces compound. When the second or third product needed the same infrastructure, the cost should be near-zero. The structure would already exist, the access policies already reasoned about and the retention logic existing.",[11,1536,1537],{},"I wanted to reuse the shape I had already committed to without building new infrastructure for a new product.",[11,1539,1540],{},"I am building around invariants, not specifics. This isn’t storage for NavEngine. It’s storage for products that require versioning, integrity and retrievability from unknown environments.",[1383,1542,1544],{"id":1543},"the-bucket-that-became-infrastructure","The Bucket That Became Infrastructure",[11,1546,1547],{},"At some point the bucket stopped being a storage decision and became infrastructure.",[11,1549,1550],{},"Storage holds data. Infrastructure carries consequences.",[11,1552,1553],{},"For this use, I went for Garage - a self-hostable and distributed S3 store. Migrations, such as our port from v2 to v3 no longer required moving 60GB+ of assets. The problem reduced to managing references - ASCII text pointing to stable objects.",[212,1555,1556,1561],{},[11,1557,1558,1560],{},[217,1559,219],{},"\nIn the case of migration, moving from v2 to v3 meant that while the database rows were ported, pointing the assets for each organisation including white-labelled data was a manual process. It relied on knowing the file system storage path and identifying what asset belonged to what organisation.",[11,1562,1563],{},"This meant moving large files over a network from server A to server B - a process I would not like to do again.",[11,1565,1566],{},"Three jobs, one system - not because it was the cleanest architecture, but because the jobs turned out to share more than I initially gave them credit for -versioning, integrity, global access, and durability beyond any single system.",[11,1568,1569],{},"What I built was, inadvertently, the distribution layer for the company's technical output.",[11,1571,1572],{},"It holds what we ship, what we back up and what we might ship next.",[11,1574,1575],{},"That is more than I planned for when I sat down to store some files.",[194,1577],{},[11,1579,1580,1581,1586,1587,182],{},"I later ported the same S3 infrastructure to my ",[48,1582,1585],{"href":1583,"rel":1584},"https://www.marvinkweyu.net/indulge/series/infrastructure",[52],"Homelabbing series",", extending it with POSIX-compliant storage via ",[48,1588,1591],{"href":1589,"rel":1590},"https://github.com/versity/versitygw",[52],"versity",[11,1593,1594,1595,1600],{},"This means that even as I develop software for ",[48,1596,1599],{"href":1597,"rel":1598},"https://www.marvinkweyu.net/indulge/owing-your-infrastructure",[52],"my own use case",", I can safely extend anything pre-existing to use the volumes I already have. That's it. No config. No workaround. And most importantly, no downtime.",[194,1602],{},[1383,1604,1606],{"id":1605},"what-this-changes","What This Changes",[11,1608,1609],{},"As I dive deeper into product agnostic infrastructure and look back at what this quarter has meant, my thinking about distribution has shifted. Uptime isn’t just about keeping systems running. It’s about designing for the moment you lose control of them - backups, extensibility, storage.",[11,1611,1612],{},"The CI/CD piece ended with designing for absence - for the reality that once the software leaves your hands, you have no control over what happens to it. The S3 structure is the complementary question: what must never leave your control?",[11,1614,1615],{},"Artifacts. Backups. Release history - the source of truth.",[11,1617,1618],{},"What the customer runs is a copy. If the copy breaks, you come back to the source. If the source is solid, the copy is recoverable.",[11,1620,1621],{},"That is the job the bucket is actually doing.",[11,1623,1624],{},"Not storage. Infrastructure.",{"title":136,"searchDepth":137,"depth":137,"links":1626},[1627,1628,1629,1630,1631,1632],{"id":1385,"depth":137,"text":1386},{"id":1419,"depth":137,"text":1420},{"id":1466,"depth":137,"text":1467},{"id":1508,"depth":137,"text":1509},{"id":1543,"depth":137,"text":1544},{"id":1605,"depth":137,"text":1606},"buckets/cover.png","2026-04-30","The infrastructure you plan and the infrastructure you end up with are rarely the same thing.",{},"/indulge/one-bucket-three-jobs","---\ntitle: One Bucket, Three Jobs\ncoverImage: buckets/cover.png\ndate: 2026-04-30\ndescription: The infrastructure you plan and the infrastructure you end up with are rarely the same thing.\ndraft: false\nfeatured: true\ngroup: Distributed Systems Field Notes\ninitial: false\nreadTime: 7\nseries: true\nseriesCover: false\nsubtitle: The Bucket That Became Infrastructure\ntags:\n  - distributed systems\n  - infrastructure\n  - s3\n  - product engineering\n  - self-hosted\n---\n\nThe [last piece](https://www.marvinkweyu.net/indulge/continuous-integration-for-licensed-software) ended with an image leaving my hands and landing somewhere I cannot see. I described shipping NavEngine as pushing a container image to a registry and trusting that a process on a customer's server will eventually notice.\n\nThat is accurate. It is also incomplete.\n\nSo far, there are two states for NavEngine; the hosted version - whose database I am responsible for whether it fails at three in the afternoon or three in the morning and the one a customer installs.\n\nFor this read, I share the story of what I built to hold it all together and why it ended up being more than a place to put files.\n\n## The Decision I Didn't Know I Was Making\n\nI had a lot of questions at the start of the project.\n\n- How would the customers download the product?\n- Where would it be stored?\n- How do companies like Canonical handle distribution?\n- What does [distribution](https://www.marvinkweyu.net/indulge/business-minded-development) actually imply?\n\nI had set out to solve a storage problem. I needed a place that was reliable, versioned, and reachable from anywhere I needed to reach it from. S3 was the obvious answer. Most things are obvious in retrospect. I did not think much about the decision at the time.\n\nWhat I did not anticipate was that a storage decision is also an architecture decision, and an architecture decision at the distribution layer has a way of spreading.\n\n## Job One: The Product Suite\n\nNavEngine started as an ISO. Before a customer would run anything on their infrastructure, they needed to have an existing artifact . They needed somewhere they could fetch it from, with integrity guarantees, across whatever network conditions exist on their end.\n\n> **Indulge**\n> This raised a different constraint: multiple products would need the same guarantees.\n\nI needed a place to store iterations as I built towards UAT - User Acceptance Testing. The registry handled images. The bucket handled everything else.\n\nThe decision to version the artifacts explicitly - not overwrite, not float - came from the same reasoning behind the floating tag in the update pipeline: I need to know exactly what was shipped.\n\n::div{.max-w-2xl.mx-auto.text-center.text-xs}\n![forbidden](/buckets/iteration.jpeg)_Versioning, but without guarantees.._\n::\n\nIf `v4.5.0` ever needed to be reproduced or rolled back to, it should already exist. Not rebuilt. Not reconstructed. Present.\n\n> **Indulge** _The hardest part of distributing software you do not host is preserving the integrity of what you shipped. Tags drift. References move. The artifact you think you gave a customer can diverge from what they actually received. Versioned, immutable storage is the only reliable paper trail when something breaks on a system you cannot access._\n\n## Job Two: The Backup\n\nWe initially served backups of NavEngine by running weekly and monthly VM snapshots. It was tedious, human-dependent and error-prone.\n\n> **Example:**\n> If an engineer needed to manually update a database record, they would have to manually create a backup for that before that individual operation. This backup would then live on the same machine.\n>\n> This meant that while the backup was present, if anything were to hit the fan, it was game over.\n>\n> _I have been here. The only grace I had was that it was on the staging environment. The story of how I got back is for another day._\n\nThe issue wasn’t that we lacked backups. It was that restoring them required the same precision that introduced the risk.\n\nA backup that lives in a different system than your product artifacts is a backup that gets forgotten about during an incident, when you are moving fast and your attention is already split. Having everything in one place means one set of access credentials, one retention policy to reason about, one place to look when something is wrong.\n\nIt was anything but elegant. It was reliable.\n\n> **Indulge**\n> Backup strategies fail at the moment of failure, not at the moment of setup. The question to ask is not \"do I have backups\" but \"can I restore from them at two in the morning, under pressure, with the right people asking questions?\"\n>\n> If the answer is uncertain, the backup strategy is incomplete regardless of how often the job runs.\n\n## Job Three: The Agnostic Product Registry\n\nThis is the job I had not planned for.\n\n> _What if we had other products to ship that had the same constraints?_\n\nBeyond hosting the versioned images of NavEngine, the storage option needed to be agnostic enough to store other images while retaining the ability to version those products too.\n\nSomewhere in the process of building the first two jobs, I noticed that the structure I was laying down did not have to be NavEngine-specific. The bucket had opinions - a versioning scheme, an artifact layout, a naming convention - but those opinions were not tied to NavEngine. They were tied to the idea of a distributable product. Any product that needed the same things NavEngine needs - versioned artifacts, clean separation between releases, a retrievable history - could sit in the same bucket structure without the structure caring what the product was.\n\nI did not set out to build a product registry. I set out to store some files. The registry emerged from the constraints.\n\nAs the team builds BusinessAI, the need to account for portability became obvious. Different pricing. Different licensing. Same distribution problem.\n\nThis matters because distribution surfaces compound. When the second or third product needed the same infrastructure, the cost should be near-zero. The structure would already exist, the access policies already reasoned about and the retention logic existing.\n\nI wanted to reuse the shape I had already committed to without building new infrastructure for a new product.\n\nI am building around invariants, not specifics. This isn’t storage for NavEngine. It’s storage for products that require versioning, integrity and retrievability from unknown environments.\n\n## The Bucket That Became Infrastructure\n\nAt some point the bucket stopped being a storage decision and became infrastructure.\n\nStorage holds data. Infrastructure carries consequences.\n\nFor this use, I went for Garage - a self-hostable and distributed S3 store. Migrations, such as our port from v2 to v3 no longer required moving 60GB+ of assets. The problem reduced to managing references - ASCII text pointing to stable objects.\n\n> **Indulge**\n> In the case of migration, moving from v2 to v3 meant that while the database rows were ported, pointing the assets for each organisation including white-labelled data was a manual process. It relied on knowing the file system storage path and identifying what asset belonged to what organisation.\n>\n> This meant moving large files over a network from server A to server B - a process I would not like to do again.\n\nThree jobs, one system - not because it was the cleanest architecture, but because the jobs turned out to share more than I initially gave them credit for -versioning, integrity, global access, and durability beyond any single system.\n\nWhat I built was, inadvertently, the distribution layer for the company's technical output.\n\nIt holds what we ship, what we back up and what we might ship next.\n\nThat is more than I planned for when I sat down to store some files.\n\n---\n\nI later ported the same S3 infrastructure to my [Homelabbing series](https://www.marvinkweyu.net/indulge/series/infrastructure), extending it with POSIX-compliant storage via [versity](https://github.com/versity/versitygw).\n\nThis means that even as I develop software for [my own use case](https://www.marvinkweyu.net/indulge/owing-your-infrastructure), I can safely extend anything pre-existing to use the volumes I already have. That's it. No config. No workaround. And most importantly, no downtime.\n\n---\n\n## What This Changes\n\nAs I dive deeper into product agnostic infrastructure and look back at what this quarter has meant, my thinking about distribution has shifted. Uptime isn’t just about keeping systems running. It’s about designing for the moment you lose control of them - backups, extensibility, storage.\n\nThe CI/CD piece ended with designing for absence - for the reality that once the software leaves your hands, you have no control over what happens to it. The S3 structure is the complementary question: what must never leave your control?\n\nArtifacts. Backups. Release history - the source of truth.\n\nWhat the customer runs is a copy. If the copy breaks, you come back to the source. If the source is solid, the copy is recoverable.\n\nThat is the job the bucket is actually doing.\n\nNot storage. Infrastructure.\n",{"title":1361,"description":1635},"indulge/one-bucket-three-jobs",[159,561,1642,160,522],"s3","il-evAaRQ3l0F-b2Rph5qz4L5r_5Qiw66KF_HEASb60",{"id":1645,"title":1646,"body":1647,"coverImage":152,"date":1744,"description":1745,"draft":146,"extension":147,"featured":146,"group":551,"initial":146,"meta":1746,"navigation":150,"path":1747,"rank":152,"rawbody":1748,"readTime":536,"seo":1749,"series":150,"seriesCover":146,"seriesOrder":152,"seriesSlug":152,"slug":152,"stem":1750,"subtitle":1751,"tags":1752,"__hash__":1753},"indulge/indulge/owing-your-infrastructure.md","Owning Your Infrastructure",{"type":8,"value":1648,"toc":1741},[1649,1652,1663,1666,1669,1677,1697,1704,1707,1714,1717,1719,1723],[11,1650,1651],{},"I have self-hosted a number of things so far , including photos, office, docker registry and chat applications for mine and my own use. This piece is a reflection on that journey, partly inspired by a recent incident where a user was banned from GitHub.",[56,1653,1655],{"className":1654},[59,60,61,62],[11,1656,1657,1661],{},[66,1658],{"alt":1659,"src":1660},"Celeste gets banned","https://res.cloudinary.com/dlxhllkxl/image/upload/v1761325142/Indulge/celeste_ban_xnxqkz.png",[21,1662,1659],{},[11,1664,1665],{},"I started self-hosting because I was bored and wanted to do hard things. Not just any hard thing but hard things that I understood ins and outs of. I kicked off with the usual simple applications, went of to projects I actually built and use and started getting dirty with writing programs for my own router - go figure. Be as it may, I have always been a tinkerer and labbing as well as my recent work with DevOps has given me the freedom to do so. What a glorious adventure.",[11,1667,1668],{},"Everything works other than my password manager. Why? Because we all know that if the server that manages your passwords goes down then the world is truly ending.",[212,1670,1671],{},[11,1672,1673,1676],{},[217,1674,1675],{},"Indulge:","\nYou cannot claim rights to what you do not actually own. Or as FUTO put it:\n'If you can’t review the source code, it’s not your software. If you can’t host the service yourself, it’s not really yours.'",[11,1678,1679,1680,965,1685,1690,1691,1696],{},"While Celeste, tweet above, was rightly banned because of his ",[48,1681,1684],{"href":1682,"rel":1683},"https://github.com/torvalds/linux/pull/1355",[52],"spam pr",[48,1686,1689],{"href":1687,"rel":1688},"https://x.com/karthihegde/status/1978316798090006532/photo/1",[52],"original title","), it brings a number of things to question. What does it mean for our convenient single signons? What happens , as with the case of ",[48,1692,1695],{"href":1693,"rel":1694},"https://www.nytimes.com/2022/08/21/technology/google-surveillance-toddler-photo.html",[52],"Mark",", if your Google account or any other SSO account is banned either temporarily or permanently? What happens to the ecosystems we so willingly give all our information to? For context, Mark's Google account was flagged when he took photos of his naked toddler for the doctor and Google mistakenly identified him as a criminal. He never recovered that account.",[212,1698,1699],{},[11,1700,1701,1703],{},[217,1702,1675],{},"\nWhat percentage of your life is dependent on your online account?",[11,1705,1706],{},"We place ourselves in bubbles of 'I cannot get banned' until it finally happens. One compromise to your online account , be it LinkedIn (as is so common these days), or any other and a cascading train of events follows.",[212,1708,1709],{},[11,1710,1711,1713],{},[217,1712,1675],{}," Maybe some of your online accounts should be your backup instead of your primary source.",[11,1715,1716],{},"I will clone my most important work to my Gitea instance over the weekend and possibly hook it to Authentik. I leave this piece here, until our next conversation - UNRAID.",[194,1718],{},[40,1720,1722],{"id":1721},"reference-notes","Reference notes",[1391,1724,1725,1733],{},[1394,1726,1727,1732],{},[48,1728,1731],{"href":1729,"rel":1730},"https://www.youtube.com/watch?v=u_Lxkt50xOg",[52],"PwediePie starts self-hosting"," to cut on subscriptions",[1394,1734,1735,1740],{},[48,1736,1739],{"href":1737,"rel":1738},"https://www.youtube.com/watch?v=zBnDWSvaQ1I",[52],"Our microphones"," and our privacy.",{"title":136,"searchDepth":137,"depth":137,"links":1742},[1743],{"id":1721,"depth":140,"text":1722},"2025-10-31","Thinking privacy, security and redundancy",{},"/indulge/owing-your-infrastructure","---\ntitle: Owning Your Infrastructure\ndate: 2025-10-31\nseries: true\ndescription: Thinking privacy, security and redundancy\ngroup: Infrastructure\nsubtitle: Our real-estate in the online space\ntags: \n  - homelabbing\n---\n\nI have self-hosted a number of things so far , including photos, office, docker registry and chat applications for mine and my own use. This piece is a reflection on that journey, partly inspired by a recent incident where a user was banned from GitHub.\n\n::div{.max-w-2xl.mx-auto.text-center.text-xs}\n![Celeste gets banned](https://res.cloudinary.com/dlxhllkxl/image/upload/v1761325142/Indulge/celeste_ban_xnxqkz.png)_Celeste gets banned_\n::\n\nI started self-hosting because I was bored and wanted to do hard things. Not just any hard thing but hard things that I understood ins and outs of. I kicked off with the usual simple applications, went of to projects I actually built and use and started getting dirty with writing programs for my own router - go figure. Be as it may, I have always been a tinkerer and labbing as well as my recent work with DevOps has given me the freedom to do so. What a glorious adventure.\n\nEverything works other than my password manager. Why? Because we all know that if the server that manages your passwords goes down then the world is truly ending.\n\n> **Indulge:**\n> You cannot claim rights to what you do not actually own. Or as FUTO put it:\n> 'If you can’t review the source code, it’s not your software. If you can’t host the service yourself, it’s not really yours.'\n\nWhile Celeste, tweet above, was rightly banned because of his [spam pr](https://github.com/torvalds/linux/pull/1355) ([original title](https://x.com/karthihegde/status/1978316798090006532/photo/1)), it brings a number of things to question. What does it mean for our convenient single signons? What happens , as with the case of [Mark](https://www.nytimes.com/2022/08/21/technology/google-surveillance-toddler-photo.html), if your Google account or any other SSO account is banned either temporarily or permanently? What happens to the ecosystems we so willingly give all our information to? For context, Mark's Google account was flagged when he took photos of his naked toddler for the doctor and Google mistakenly identified him as a criminal. He never recovered that account.\n\n> **Indulge:**\n> What percentage of your life is dependent on your online account?\n\nWe place ourselves in bubbles of 'I cannot get banned' until it finally happens. One compromise to your online account , be it LinkedIn (as is so common these days), or any other and a cascading train of events follows.\n\n> **Indulge:** Maybe some of your online accounts should be your backup instead of your primary source.\n\nI will clone my most important work to my Gitea instance over the weekend and possibly hook it to Authentik. I leave this piece here, until our next conversation - UNRAID.\n\n---\n\n### Reference notes\n\n\n- [PwediePie starts self-hosting](https://www.youtube.com/watch?v=u_Lxkt50xOg) to cut on subscriptions\n\n- [Our microphones](https://www.youtube.com/watch?v=zBnDWSvaQ1I) and our privacy.\n",{"title":1646,"description":1745},"indulge/owing-your-infrastructure","Our real-estate in the online space",[1357],"QdRG7sXGg6vS9oix5N5Ne5T0ZVym_RTiJMvr0KOPZHo",{"id":1755,"title":1756,"body":1757,"coverImage":1796,"date":1797,"description":1798,"draft":146,"extension":147,"featured":146,"group":1799,"initial":146,"meta":1800,"navigation":150,"path":1801,"rank":152,"rawbody":1802,"readTime":536,"seo":1803,"series":150,"seriesCover":146,"seriesOrder":152,"seriesSlug":152,"slug":152,"stem":1804,"subtitle":1805,"tags":1806,"__hash__":1812},"indulge/indulge/research-and-build-do-not-compose.md","Research and Build Don't Compose",{"type":8,"value":1758,"toc":1794},[1759,1762,1765,1768,1785,1788,1791],[11,1760,1761],{},"Research and product development are usually presented as sequential: you study something, then you build something. In practice, at least in my experience, they run in parallel and frequently interfere with each other in ways that are neither clean nor comfortable.",[11,1763,1764],{},"Research & Building is a series about what happens at that boundary. It documents the experience of doing graduate research in ecological niche modelling of foot-and-mouth disease under climate change while simultaneously building and shipping commercial software. Sometimes those two things reinforce each other. Often they compete for the same hours, the same cognitive load, and the same intellectual energy.",[11,1766,1767],{},"The series covers questions I haven't found well-answered elsewhere:",[1391,1769,1770,1773,1776,1779,1782],{},[1394,1771,1772],{},"How do you maintain research rigour when a client deadline is three days away?",[1394,1774,1775],{},"What does it mean to treat your own infrastructure as a research instrument?",[1394,1777,1778],{},"When does a paper become a positioning document, and is that a problem?",[1394,1780,1781],{},"How do you read academic literature as a practitioner without losing the parts that actually matter for building things?",[1394,1783,1784],{},"What does the practitioner-researcher split cost you - and what does it give you that neither role has alone?",[11,1786,1787],{},"Some pieces are reading notes from books and papers that shaped how I think. Others are field reports from the overlap: moments where a statistical model informed an architectural decision, or a production failure reframed a research question. A few will be honest accounts of getting both wrong at once.",[11,1789,1790],{},"This is not a series about work-life balance or productivity systems. It's about what it actually looks like to operate across two disciplines that rarely acknowledge each other - and what becomes possible when they do.",[11,1792,1793],{},"If you're a researcher who builds things, or an engineer who reads papers and wonders whether they're allowed to, these notes are for you. Nobody warned us the overlap would be this loud.",{"title":136,"searchDepth":137,"depth":137,"links":1795},[],"research/research_and_build.jpeg","2026-05-29","What it looks like to run ecological models and ship licensed software in the same week - and what each discipline takes from the other.","Research & Building",{},"/indulge/research-and-build-do-not-compose","---\ntitle: Research and Build Don't Compose\ndate: 2026-05-29\ndescription: What it looks like to run ecological models and ship licensed software in the same week - and what each discipline takes from the other.\ncoverImage: research/research_and_build.jpeg\ndraft: false\nfeatured: false\ngroup: Research & Building\ninitial: false\nreadTime: 5\nseries: true\nseriesCover: false\nsubtitle: On operating across two disciplines that rarely acknowledge each other\ntags:\n  - research\n  - software-engineering\n  - geospatial\n  - climate\n  - craft\n---\n\nResearch and product development are usually presented as sequential: you study something, then you build something. In practice, at least in my experience, they run in parallel and frequently interfere with each other in ways that are neither clean nor comfortable.\n\nResearch & Building is a series about what happens at that boundary. It documents the experience of doing graduate research in ecological niche modelling of foot-and-mouth disease under climate change while simultaneously building and shipping commercial software. Sometimes those two things reinforce each other. Often they compete for the same hours, the same cognitive load, and the same intellectual energy.\n\nThe series covers questions I haven't found well-answered elsewhere:\n\n- How do you maintain research rigour when a client deadline is three days away?\n- What does it mean to treat your own infrastructure as a research instrument?\n- When does a paper become a positioning document, and is that a problem?\n- How do you read academic literature as a practitioner without losing the parts that actually matter for building things?\n- What does the practitioner-researcher split cost you - and what does it give you that neither role has alone?\n\nSome pieces are reading notes from books and papers that shaped how I think. Others are field reports from the overlap: moments where a statistical model informed an architectural decision, or a production failure reframed a research question. A few will be honest accounts of getting both wrong at once.\n\nThis is not a series about work-life balance or productivity systems. It's about what it actually looks like to operate across two disciplines that rarely acknowledge each other - and what becomes possible when they do.\n\nIf you're a researcher who builds things, or an engineer who reads papers and wonders whether they're allowed to, these notes are for you. Nobody warned us the overlap would be this loud.\n",{"title":1756,"description":1798},"indulge/research-and-build-do-not-compose","On operating across two disciplines that rarely acknowledge each other",[1807,1808,1809,1810,1811],"research","software-engineering","geospatial","climate","craft","3AjekfpRpnklMgiwKp-_0xte7mduJcGYKvNG74Uu4xo",{"id":1814,"title":1799,"body":1815,"coverImage":152,"date":531,"description":1819,"draft":146,"extension":147,"featured":146,"group":1799,"initial":146,"meta":1820,"navigation":150,"path":1821,"rank":152,"rawbody":1822,"readTime":536,"seo":1823,"series":150,"seriesCover":150,"seriesOrder":152,"seriesSlug":152,"slug":152,"stem":1824,"subtitle":1825,"tags":1826,"__hash__":1829},"indulge/indulge/research-and-building.md",{"type":8,"value":1816,"toc":1817},[],{"title":136,"searchDepth":137,"depth":137,"links":1818},[],"Field notes from the boundary between research and product. What it looks like to run ecological models and ship licensed software in the same week.",{},"/indulge/research-and-building","---\ntitle: Research & Building\nseries: true\nseriesCover: true\ngroup: Research & Building\ndescription: Field notes from the boundary between research and product. What it looks like to run ecological models and ship licensed software in the same week.\nfeatured: false\ndraft: false\nsubtitle: Science & Product\ntags:\n  - research\n  - product\n  - geospatial\n  - machine learning\n---\n",{"title":1799,"description":1819},"indulge/research-and-building","Science & Product",[1807,1827,1809,1828],"product","machine learning","JDWsMO7V0CDFeVlu9h9gIgPNTPuCip1Dfh6noowMzqM",{"id":1831,"title":1832,"body":1833,"coverImage":152,"date":2008,"description":2009,"draft":146,"extension":147,"featured":146,"group":2010,"initial":146,"meta":2011,"navigation":150,"path":2012,"rank":152,"rawbody":2013,"readTime":536,"seo":2014,"series":150,"seriesCover":146,"seriesOrder":152,"seriesSlug":152,"slug":152,"stem":2015,"subtitle":2016,"tags":2017,"__hash__":2019},"indulge/indulge/the-20%-that-makes-all-the-difference.md","The 20% that makes all the difference",{"type":8,"value":1834,"toc":2002},[1835,1840,1843,1852,1855,1860,1869,1873,1876,1881,1884,1893,1895,1907,1915,1924,1932,1941,1944,1950,1954,1963,1965,1967,1969,1972],[212,1836,1837],{},[11,1838,1839],{},"“And I took the road less travelled. And that made all the difference.”",[11,1841,1842],{},"Time management is something I find myself working through every once in a while. This will often involve writing down weekly items to check off, crossing 2 of the twenty of my to-do list, crossing a 3/4 or throwing it all together.",[11,1844,1845,1846,1851],{},"My read on effective engineering dubbed, ",[48,1847,1850],{"href":1848,"rel":1849},"https://www.goodreads.com/book/show/25238425-the-effective-engineer?ac=1&from_search=true&qid=V59d6Adpga&rank=1",[52],"The Effective Enginee","r by Edmond Lau, has been as inspirational as it has been eye-opening. From this, a few thought processes came up that would be worth noting, specifically regarding impact, sustainability and effectiveness.",[11,1853,1854],{},"To quote:",[212,1856,1857],{},[11,1858,1859],{},"I worked the long hours because I wanted to make a meaningful impact, but I couldn’t help but wonder: Was putting in 70-80 hour weeks really the most effective way of ensuring our startup’s success? Our intentions were sound, but could we have worked smarter? - Edmond Lau",[11,1861,1862,1863,1868],{},"With this piece, and in a builder’s pursuit of mastery, I will interweave a few ideas, thoughts and research points to challenge convention and support the contrarian or vice versa. It is a leap out of the ",[48,1864,1867],{"href":1865,"rel":1866},"https://www.marvinkweyu.net/indulge/work_and_life",[52],"day-to-day routine"," of the builder and into the vastness unknown.",[40,1870,1872],{"id":1871},"the-always-be-shipping-mantra","The Always Be Shipping mantra",[11,1874,1875],{},"The most captivating moment from the book, at least for me, was the mention and talk of experimentation. The goal being to build not for the sake of it. I call it flow and experimentation.",[212,1877,1878],{},[11,1879,1880],{},"I can’t tell you why you need a home computer right now. I mean, people ask me, “Why should I buy a computer in my home?” And I say, “Well, to learn about it, to run some fun simulations. If you’ve got some kids, they should probably know about it in terms of literacy. They can probably get some good educational software, especially if they’re younger. “You can hook up to the source and, you know, do whatever you’re going to do. Meet women, I don’t know. But other than that, there’s no good reason to buy one for your house right now. But there will be. There will be.” - Steve Jobs (Speech at the International Design Conference in Aspen, June 1983)",[11,1882,1883],{},"In the early 2000s, Google pioneered what was called “The 20% time” - a chip of time within which engineers in its domain would be free to leverage Google’s technology to build their own products unhindered. Suffice it to say, this led to the development of products such as Google News, Google Maps and Gmail.",[11,1885,1886,1887,1892],{},"Consequently, this fostered a culture of innovation and creativity which contributed to its relish within the engineering community as the ‘go-to’ company to work with. ",[48,1888,1891],{"href":1889,"rel":1890},"https://www.wired.com/2012/12/llinkedin-20-percent-time/",[52],"LinkedIn, Atlassian and Adobe"," followed suit with hopes of catching up.",[194,1894],{},[56,1896,1898],{"className":1897},[59,60,61,62],[11,1899,1900,1904],{},[66,1901],{"alt":1902,"src":1903},"Instagram","https://res.cloudinary.com/dlxhllkxl/image/upload/v1687601656/Indulge/how-instagram-started_xnrhsa.png",[21,1905,1906],{},"How Instagram Started",[212,1908,1909],{},[11,1910,1911,1912],{},"I don't know what the future will bring, but if you stop investing in basic research today, you won't have a future. - ",[217,1913,1914],{},"Neil deGrasse Tyson",[11,1916,1917,1918,1923],{},"From my experience, it is always the builders that do more than is within their scope that end up being the great men and women in their domain; ",[48,1919,1922],{"href":1920,"rel":1921},"https://www.marvinkweyu.net/indulge/a_call_to_dream",[52],"the innovators of the future",". To mention a few of these; ProductHunt, Slack, Twitter, Pinterest (hello Elle), Shopify, AppSumo and GitHub. For the founders, it was the little something on the side that made all the difference.",[11,1925,1926,1927,182],{},"Remarkably, by the time the aforementioned companies were either acquired, went public or reached a certain user-base marking, their ratio of engineer to user was in the scale of one to thousands or even millions. In other words, they had built successful enterprises not with an abundance of human resources but with a vision for a product that ",[48,1928,1931],{"href":1929,"rel":1930},"https://www.marvinkweyu.net/indulge/necessity_versus_opportunity",[52],"addressed their immediate need",[212,1933,1934],{},[11,1935,1936,1937,1940],{},"The reason we ",[626,1938,1939],{},"Woz and I"," built a computer was that we wanted one, and we couldn’t afford to buy one… We were just two teenagers. We started trying to build them and scrounging parts around Silicon Valley where we could…All our friends wanted them too. It was taking up all of our spare time because our friends were not that skilled at building them, so Woz and I were building them for them. - Steve Jobs, 1996, Apple 20th anniversary.",[1942,1943],"br",{},[11,1945,1946,1949],{},[217,1947,1948],{},"Thought:","\nIf someone does not think you are insane for how relentlessly you work to achieve your dreams, you might want to re-evaluate your choices. Being called crazy and being rejected is not such a bad thing.",[1383,1951,1953],{"id":1952},"conclusion","Conclusion",[11,1955,1956,1957,1962],{},"In the end, ",[48,1958,1961],{"href":1959,"rel":1960},"https://youtu.be/pqQrL1K0Z5g?t=184",[52],"you might never know when or where your project might be of use"," to the rest of the populous. The least you can do, however, is to take a chunk of time within your day to build, innovate and imagine the possibilities.",[194,1964],{},[1942,1966],{},[40,1968,1722],{"id":1721},[11,1970,1971],{},"For some notes on how some of the products above came to be, I leave some starter resources here.",[1391,1973,1974,1981,1988,1995],{},[1394,1975,1976],{},[48,1977,1980],{"href":1978,"rel":1979},"https://nira.com/github-history/",[52],"How GitHub Democratized Coding, Built a $2 Billion Business, and Found a New Home at Microsoft",[1394,1982,1983],{},[48,1984,1987],{"href":1985,"rel":1986},"https://www.youtube.com/watch?v=2rqwi63Q1Gs",[52],"Steve Jobs Speech at the International Design Conference 1983",[1394,1989,1990],{},[48,1991,1994],{"href":1992,"rel":1993},"https://medium.com/lets-make-things/the-origin-of-product-hunt-7acb09e2593a",[52],"The origin of ProductHunt",[1394,1996,1997],{},[48,1998,2001],{"href":1999,"rel":2000},"https://www.goodreads.com/book/show/128533513-make-something-wonderful",[52],"Make Something Wonderful - Steve Jobs",{"title":136,"searchDepth":137,"depth":137,"links":2003},[2004,2005],{"id":1871,"depth":140,"text":1872},{"id":1952,"depth":137,"text":1953,"children":2006},[2007],{"id":1721,"depth":140,"text":1722},"2024-08-28","Thoughts on time, innovation and self","Book Reviews",{},"/indulge/the-20percent-that-makes-all-the-difference","---\ntitle: The 20% that makes all the difference\nsubtitle: Side projects and their impact on culture and innovation\ndescription: Thoughts on time, innovation and self\ngroup: Book Reviews\ninitial: false\nseries: true\nseriesCover: false\ndate: 2024-08-28\ntags:\n  - book review\n---\n\n> “And I took the road less travelled. And that made all the difference.”\n\nTime management is something I find myself working through every once in a while. This will often involve writing down weekly items to check off, crossing 2 of the twenty of my to-do list, crossing a 3/4 or throwing it all together.\n\nMy read on effective engineering dubbed, [The Effective Enginee](https://www.goodreads.com/book/show/25238425-the-effective-engineer?ac=1&from_search=true&qid=V59d6Adpga&rank=1)r by Edmond Lau, has been as inspirational as it has been eye-opening. From this, a few thought processes came up that would be worth noting, specifically regarding impact, sustainability and effectiveness.\n\nTo quote:\n\n> I worked the long hours because I wanted to make a meaningful impact, but I couldn’t help but wonder: Was putting in 70-80 hour weeks really the most effective way of ensuring our startup’s success? Our intentions were sound, but could we have worked smarter? - Edmond Lau\n\nWith this piece, and in a builder’s pursuit of mastery, I will interweave a few ideas, thoughts and research points to challenge convention and support the contrarian or vice versa. It is a leap out of the [day-to-day routine](https://www.marvinkweyu.net/indulge/work_and_life) of the builder and into the vastness unknown.\n\n### The Always Be Shipping mantra\n\nThe most captivating moment from the book, at least for me, was the mention and talk of experimentation. The goal being to build not for the sake of it. I call it flow and experimentation.\n\n> I can’t tell you why you need a home computer right now. I mean, people ask me, “Why should I buy a computer in my home?” And I say, “Well, to learn about it, to run some fun simulations. If you’ve got some kids, they should probably know about it in terms of literacy. They can probably get some good educational software, especially if they’re younger. “You can hook up to the source and, you know, do whatever you’re going to do. Meet women, I don’t know. But other than that, there’s no good reason to buy one for your house right now. But there will be. There will be.” - Steve Jobs (Speech at the International Design Conference in Aspen, June 1983)\n\nIn the early 2000s, Google pioneered what was called “The 20% time” - a chip of time within which engineers in its domain would be free to leverage Google’s technology to build their own products unhindered. Suffice it to say, this led to the development of products such as Google News, Google Maps and Gmail.\n\nConsequently, this fostered a culture of innovation and creativity which contributed to its relish within the engineering community as the ‘go-to’ company to work with. [LinkedIn, Atlassian and Adobe](https://www.wired.com/2012/12/llinkedin-20-percent-time/) followed suit with hopes of catching up.\n\n---\n\n::div{.max-w-2xl.mx-auto.text-center.text-xs}\n![Instagram](https://res.cloudinary.com/dlxhllkxl/image/upload/v1687601656/Indulge/how-instagram-started_xnrhsa.png)_How Instagram Started_\n::\n\n> I don't know what the future will bring, but if you stop investing in basic research today, you won't have a future. - **Neil deGrasse Tyson**\n\nFrom my experience, it is always the builders that do more than is within their scope that end up being the great men and women in their domain; [the innovators of the future](https://www.marvinkweyu.net/indulge/a_call_to_dream). To mention a few of these; ProductHunt, Slack, Twitter, Pinterest (hello Elle), Shopify, AppSumo and GitHub. For the founders, it was the little something on the side that made all the difference.\n\nRemarkably, by the time the aforementioned companies were either acquired, went public or reached a certain user-base marking, their ratio of engineer to user was in the scale of one to thousands or even millions. In other words, they had built successful enterprises not with an abundance of human resources but with a vision for a product that [addressed their immediate need](https://www.marvinkweyu.net/indulge/necessity_versus_opportunity).\n\n> The reason we [Woz and I] built a computer was that we wanted one, and we couldn’t afford to buy one… We were just two teenagers. We started trying to build them and scrounging parts around Silicon Valley where we could…All our friends wanted them too. It was taking up all of our spare time because our friends were not that skilled at building them, so Woz and I were building them for them. - Steve Jobs, 1996, Apple 20th anniversary.\n\n:br\n\n**Thought:**\nIf someone does not think you are insane for how relentlessly you work to achieve your dreams, you might want to re-evaluate your choices. Being called crazy and being rejected is not such a bad thing.\n\n## Conclusion\n\nIn the end, [you might never know when or where your project might be of use](https://youtu.be/pqQrL1K0Z5g?t=184) to the rest of the populous. The least you can do, however, is to take a chunk of time within your day to build, innovate and imagine the possibilities.\n\n---\n\n:br\n\n### Reference notes\n\nFor some notes on how some of the products above came to be, I leave some starter resources here.\n\n- [How GitHub Democratized Coding, Built a $2 Billion Business, and Found a New Home at Microsoft](https://nira.com/github-history/)\n- [Steve Jobs Speech at the International Design Conference 1983](https://www.youtube.com/watch?v=2rqwi63Q1Gs)\n- [The origin of ProductHunt](https://medium.com/lets-make-things/the-origin-of-product-hunt-7acb09e2593a)\n- [Make Something Wonderful - Steve Jobs](https://www.goodreads.com/book/show/128533513-make-something-wonderful)\n",{"title":1832,"description":2009},"indulge/the-20%-that-makes-all-the-difference","Side projects and their impact on culture and innovation",[2018],"book review","p3GGD9TmmFIxEKUjzE1MeMw6JbxPqmwBhQzOBhO5ltA",{"id":2021,"title":2022,"body":2023,"coverImage":152,"date":2056,"description":2057,"draft":146,"extension":147,"featured":146,"group":551,"initial":146,"meta":2058,"navigation":150,"path":2059,"rank":152,"rawbody":2060,"readTime":536,"seo":2061,"series":150,"seriesCover":146,"seriesOrder":152,"seriesSlug":152,"slug":152,"stem":2062,"subtitle":136,"tags":2063,"__hash__":2064},"indulge/indulge/the-start-of-an-adventure.md","Homelabbing - The Start Of An Adventure",{"type":8,"value":2024,"toc":2053},[2025,2028,2030,2034,2037,2040,2050],[11,2026,2027],{},"My recent work on Scale Computing and Opennebula has piqued my interest in home lab environments. I decided to take a peek and got myself a HP G2 800 minicomputer for play.This , along with a number of smart devices to really push the limits of what is possible.",[194,2029],{},[40,2031,2033],{"id":2032},"so-what-is-a-home-lab","So, what is a home lab?",[11,2035,2036],{},"A home lab serves as a safe space for tech enthusiasts to experiment on ideas.",[11,2038,2039],{},"So far, I have been able to create and manage virtual machines including the assignment of public IP address and management of mikrotik routers. All this has been in a globally distributed environment with data centers all over. While this is exciting in and of itself, I always wondered; what doors could I possibly unlock if I ran my own set up within a space I could control? I built this space for exactly this reason.",[11,2041,1366,2042,2049],{},[48,2043,2046],{"href":2044,"rel":2045},"https://marvinkweyu.net/indulge/series/infrastructure",[52],[21,2047,2048],{},"#homelabbing"," series serves as a documentation of the experiments I perform on this handy resource I currently have. It will encompass everything from setting up , power consumption, projects I am working on to the paths I take in the foreseeable future.",[11,2051,2052],{},"It is my hope that on this wire of the internet, someone finds a spark - the same curiosity that led me down this rabbit hole. Whether you're just getting started or are years into your own home lab journey, may this series serve as a source of insight, inspiration, and maybe even a few lessons from my inevitable missteps. The adventure has just begun, and I look forward to sharing each step along the way.",{"title":136,"searchDepth":137,"depth":137,"links":2054},[2055],{"id":2032,"depth":140,"text":2033},"2024-12-12","Navigating a mental mindshif on entrepreneurship",{},"/indulge/the-start-of-an-adventure","---\ntitle: Homelabbing - The Start Of An Adventure\nseries: true\ngroup: Infrastructure\nsubtitle: \"\"\ndescription: Navigating a mental mindshif on entrepreneurship\ndate: 2024-12-12\ntags: [homelabbing]\n---\n\nMy recent work on Scale Computing and Opennebula has piqued my interest in home lab environments. I decided to take a peek and got myself a HP G2 800 minicomputer for play.This , along with a number of smart devices to really push the limits of what is possible.\n\n---\n\n### So, what is a home lab?\n\nA home lab serves as a safe space for tech enthusiasts to experiment on ideas.\n\nSo far, I have been able to create and manage virtual machines including the assignment of public IP address and management of mikrotik routers. All this has been in a globally distributed environment with data centers all over. While this is exciting in and of itself, I always wondered; what doors could I possibly unlock if I ran my own set up within a space I could control? I built this space for exactly this reason.\n\nThe [_#homelabbing_](https://marvinkweyu.net/indulge/series/infrastructure) series serves as a documentation of the experiments I perform on this handy resource I currently have. It will encompass everything from setting up , power consumption, projects I am working on to the paths I take in the foreseeable future.\n\nIt is my hope that on this wire of the internet, someone finds a spark - the same curiosity that led me down this rabbit hole. Whether you're just getting started or are years into your own home lab journey, may this series serve as a source of insight, inspiration, and maybe even a few lessons from my inevitable missteps. The adventure has just begun, and I look forward to sharing each step along the way.\n",{"title":2022,"description":2057},"indulge/the-start-of-an-adventure",[1357],"Iz0ThrPNrtbpGhmitPcHELhA6jdHGfHQNdlz-bkeEI8",{"id":2066,"title":2067,"body":2068,"coverImage":2110,"date":2111,"description":532,"draft":146,"extension":147,"featured":146,"group":148,"initial":146,"meta":2112,"navigation":150,"path":2113,"rank":152,"rawbody":2114,"readTime":536,"seo":2115,"series":150,"seriesCover":146,"seriesOrder":152,"seriesSlug":152,"slug":152,"stem":2116,"subtitle":2117,"tags":2118,"__hash__":2119},"indulge/indulge/why-distributed-systems-field-notes.md","Why These Notes Exist",{"type":8,"value":2069,"toc":2108},[2070,2073,2076,2079,2099,2102,2105],[11,2071,2072],{},"Distributed systems are rarely built in one clean leap. They evolve through compromises, deadlines, partial rewrites and decisions made with incomplete information. Documentation often captures what was built, but almost never why it was built that way- or what broke along the way.",[11,2074,2075],{},"Distributed Systems Field Notes is a collection of essays drawn from building and operating production systems across IaaS, SaaS and data-intensive platforms. I focus less on ideal architectures and more on the realities of shipping software: trade-offs, constraints and the second-order effects that only surface once users, data and failure modes arrive.",[11,2077,2078],{},"The series covers topics such as:",[1391,2080,2081,2084,2087,2090,2093,2096],{},[1394,2082,2083],{},"Service boundaries and control planes",[1394,2085,2086],{},"Containerization and OCI image packaging",[1394,2088,2089],{},"Event-driven and message-based architectures",[1394,2091,2092],{},"Multi-tenancy and isolation strategies",[1394,2094,2095],{},"Deployment models and operational simplicity",[1394,2097,2098],{},"Patterns that held up under pressure-and those that didn’t",[11,2100,2101],{},"Most examples are grounded in real products I’ve worked on, including infrastructure platforms, applied AI systems and multi-tenant SaaS. Names and details may change, but the architectural pressures remain the same.",[11,2103,2104],{},"This is not a tutorial series. You won’t find step-by-step guides or “best practices” divorced from context. Instead, these field notes aim to document decision-making in motion: what seemed reasonable at the time, what failed later and what I would do differently with hindsight.",[11,2106,2107],{},"If you’re building systems that need to survive growth, team changes, and real operational load, I hope these notes save you a few painful lessons - or at least help you recognize them earlier.",{"title":136,"searchDepth":137,"depth":137,"links":2109},[],"series/software_architecture.jpg","2026-02-06",{},"/indulge/why-distributed-systems-field-notes","---\ntitle: Why These Notes Exist\ndate: 2026-02-06\ndescription: Hard-won insights from architecting and operating systems at scale. Patterns, anti-patterns and war stories.\ncoverImage: series/software_architecture.jpg\ndraft: false\nfeatured: false\ngroup: Distributed Systems Field Notes\ninitial: false\nreadTime: 5\nseries: true\nseriesCover: false\nsubtitle: Documenting decisions, trade-offs, and second-order effects\ntags:\n  - distributed systems\n---\n\nDistributed systems are rarely built in one clean leap. They evolve through compromises, deadlines, partial rewrites and decisions made with incomplete information. Documentation often captures what was built, but almost never why it was built that way- or what broke along the way.\n\nDistributed Systems Field Notes is a collection of essays drawn from building and operating production systems across IaaS, SaaS and data-intensive platforms. I focus less on ideal architectures and more on the realities of shipping software: trade-offs, constraints and the second-order effects that only surface once users, data and failure modes arrive.\n\nThe series covers topics such as:\n\n- Service boundaries and control planes\n- Containerization and OCI image packaging\n- Event-driven and message-based architectures\n- Multi-tenancy and isolation strategies\n- Deployment models and operational simplicity\n- Patterns that held up under pressure-and those that didn’t\n\nMost examples are grounded in real products I’ve worked on, including infrastructure platforms, applied AI systems and multi-tenant SaaS. Names and details may change, but the architectural pressures remain the same.\n\nThis is not a tutorial series. You won’t find step-by-step guides or “best practices” divorced from context. Instead, these field notes aim to document decision-making in motion: what seemed reasonable at the time, what failed later and what I would do differently with hindsight.\n\nIf you’re building systems that need to survive growth, team changes, and real operational load, I hope these notes save you a few painful lessons - or at least help you recognize them earlier.\n",{"title":2067,"description":532},"indulge/why-distributed-systems-field-notes","Documenting decisions, trade-offs, and second-order effects",[159],"FkpFnrgXXyf-ryy-1LWrCDkwmyc7uuCopvXlNRzPx4g",1781087028781]