Ask stories

Yesterday my production app went down. The cause? DigitalOcean's managed PostgreSQL update broke private VPC connectivity to their managed Kubernetes.

Public endpoint worked. Private endpoint timed out. Root cause: a Cilium bug (#34503) where ARP entries go stale after infrastructure changes.

DO support responded relatively quickly (<12hrs). Their fix? Deploy a DaemonSet from a random GitHub user to ping stale ARP entries every 10 seconds. The upstream Cilium fix is merged but not yet deployed to DOKS. No ETA.

I chose managed services specifically to avoid ops emergencies. We're a tiny startup paying the premium so someone else handles this. Instead, I spent late night hours debugging VPC routing issues in a networking layer I don't control.

HN's usual advice is "just use managed services, focus on the business." Generally good advice. But managed doesn't mean worry-free, it means trading your failure modes for the vendor's failure modes. You're not choosing between problems and no problems. You're choosing between problems you control and (fewer?) problems you don't.

Still using DO. Still using managed services. Just with fewer illusions about what "managed" means.

What are you working on? Any new ideas that you're thinking about?

My Cofounder Matching inbox seems stuck in an infinite loop and won’t load any messages or matches. Tried refresh, different browsers, and re-login.

If anyone from YC is here or others are seeing this, would appreciate pointers.

It seems kinda inevitable. Some modern-day equivalent of the latter-day Kevin Mitnick is bound to be out there somewhere. Wouldn't you think?

I was trying to imagine…

• What sort of AI-related exploit is more likely to be one of the first?

• What might be the nature of a cyberattack the black hat vibe coders might be vibing up for us?

Of course, I hope I'm wrong. I hope that all the MCP servers or whatever and the other AI network infrastructure is as secure as we're told they are.

I've often been in the market for new software engineer opportunities and have found some untraditional ways of finding new jobs. But with so many job search tools, AI and ATS matching I'm curious what job search strategies are working for you today? Also, what has been a waste of time?

Last week I had two submissions reach the HN front page. Before that, I assumed the ranking followed a roughly monotonic time-decay curve once a post peaked. While watching these two posts, I noticed something different: their rank would sometimes improve after declining, and occasionally oscillate back and forth (Here is a graph showing this behavior: https://imgur.com/a/AQnSh6I). what mechanisms in the HN ranking algorithm cause this kind of non-monotonic movement. Is it due to vote weighting, comment activity, re-ranking batches, or something else?

The graph is from: https://track-hacker-news.com/

EDIT: This is the post referenced: https://news.ycombinator.com/item?id=46512881

How are you generating release notes in your projects?

I just had to backfill a year of docs release notes for an OSS repo. I ended up writing a small release tag driven generator: it walks git tags, collects merged PRs between releases, buckets them into categories, and renders Markdown/MDX grouped by year -> month -> category -> version. I also added an optional LLM step that outputs structured JSON via Pydantic schema for PR bullets that includes monthly summaries. It is idempotent and preserves manual edits or omissions, so you can auto-generate, then curate over time.

I’m curious what works for you in practice:

- Towncrier?

- reno?

- GitHub Releases / auto-generated notes?

- something else?

What do you like/hate about your current setup? Any tools you’d recommend?

If anyone wants to see the script I wrote, I am happy to share it and would love some constructive feedback.

I've self-hosted Mattermost since 2019 (originally to escape Slack’s retention limits for family chat; logistics, photos, random history). It was a huge hurdle getting them all on Slack, and then moving them again to Mattermost.

This morning, I upgraded our long-running Omnibus install from v10.9.1 → v11, following Mattermost's explicit deprecation notice[0] and recommendation to switch to standard Binary/Docker/k8s/etc deployment.

Upgrade went cleanly:

Postgres migration fine Data still present in DB Service came back up in minutes

Post-restart the UI shows: “10,000-message limit reached. Messages sent before May 16, 2023 are hidden.” Clarification: no DELETE FROM anywhere. All rows still in PostgreSQL. The limit is UI/view-layer only, retroactively applied after "upgrade".

Checked beforehand:

- Omnibus deprecation announcement [0] - v11 upgrade guide [4] - Admin console nags/links

None mentioned that an in-place upgrade would activate hard visibility caps on pre-existing data. First indication was the running instance after restart.

Currently (as of 2026-01-12_1337):

- https://mattermost.com/pl/mattermost-entry-limits (In-App Link to Limits) [1] 404s (Page not found) - https://mattermost.com/pricing/ (also an in-app link) [2] only has “Contact Sales”, no enumerated limits (or prices). - https://docs.mattermost.com/administration-guide/manage/product-limits.html [3], used Kagi to find this, no mention of message limits

I'm happy to pay for supported software. The friction is the surprise enforcement on self-hosted + self-managed DB. Upgrade path quietly turned history into a licensed feature without forewarning in the upgrade docs.

For now:

We're leaving the v11 instance up (secure, current messages flow normally). The system console and dashboard aren't loading correctly, possibly due to the message limits, possibly due to other issues.

I'm thinking some sort of rsync + static HTML archive of current visible history + DB dump for offline search. I'm afraid future policy changes could further gate access, and would love recommendations.

Posting mostly as warning to other long-running Omnibus users (small groups, families, hobby servers) who are about to upgrade and may not expect the view limit to kick in retroactively.

If your install predates the limit introduction and has >10k messages, test upgrade in staging first—or assume the docs are incomplete on enforcement timing. Don't expect to get help in the forum [5], the last post (before mine today) is from October 2025 and the moderator directed to the Gitlab Omnibus thread which doesn't provide details on message limits or answer the other questions.

Curious if others hit surprises on the v11 "upgrade", and if anyone has any suggestions.

[0] https://github.com/mattermost/mattermost-omnibus [1] https://mattermost.com/pl/mattermost-entry-limits [2] https://mattermost.com/pricing/ [3] https://docs.mattermost.com/administration-guide/manage/product-limits.html [4] https://docs.mattermost.com/deployment-guide/server/deploy-linux.html [5] https://forum.mattermost.com/t/clarification-needed-for-the-upgrade-to-v11/25344

Recently built something where simple domain-specific heuristics crushed a fancy ML approach I assumed would win. This has me thinking about how often we reach for complex tools when simpler ones would work better. Occam's razor moments.

Anyone have similar stories? Curious about cases where knowing your domain beat throwing compute at the problem.

Previously it has been using Akamai

Hey HN,

We originally were building vertical voice agents for fitness centers. During customer conversations, the idea of collecting payment during the call (via SMS link, keypad, or voice) would sometimes come up as something that might be useful.

We’re trying to understand how people have approached this in practice.

If you’ve worked on production voice systems or payments, I had a few questions: 1) Have you implemented in-call payment flows? 2) What tradeoffs or issues did you run into (compliance, reliability, etc.)? 3) Was it something customers actually used, or mostly ignored?

We also noticed a YC F24 company (Protegee) explored this space and later pivoted, which makes us curious what lessons people took away from that generation of attempts.

Appreciate any real-world experience or pointers. We’re trying to learn from folks who’ve actually shipped this.

A common one is webp. This hinders image uploading services and makes it harder to save and share the images.

Ask HN,

I’m a software engineer and founder. Over the years, I’ve built backend and cloud systems and created a few internal tools to address recurring operational issues.

Before spending time building anything new, I’m trying to understand what problems are actually common today.

So I’m curious: what’s a process in your business that still costs you time every week and shouldn’t? For example: something manual, spreadsheet-heavy, or a tool you tried and abandoned.

If you’re comfortable sharing publicly, I’d appreciate hearing about it. Thanks.

Are you coding more or less, managing people differently, or making decisions in new ways because of AI tools? Which tools (LLMs, copilots, internal agents, analytics, etc.) have meaningfully stuck, and which turned out to be hype? I’m especially interested in concrete changes to how you plan, review work, and support teams.

Hop on an international flight, remember you had set automatic downloads for your favorite podcast show. Open the app to find the latest (or even last few) episodes are not downloaded. Why is it so hard to auto download podcasts when the phone is charging at night and connected to a WiFi? What’s the point of auto-download button?

I have been mostly anti AI. I did experiment a bit with aider and free models, but my results were inconsistent, and nothing to worry about.

However, recently I have purchased the max plan from anthropic and have been vibing with Claude code since then. And wow, the results are very good. With a good enough prompt, and planning step, it could generate full features in a project with 20k LOC, with very little modifications needed by me after review.

I heard even more success stories from friends who gave Claude 3-4 different features that Claude would develop in parallel.

On top of that, everyone seems to produce side project at an astronomical rate, both among my friends, and here on HN where fully complete project that would take months to develop, seem to appear after few hours with Claude code.

So, my questions is, is programming as a profession cooked? Are most of us going to be replaced with a “supervisor” who runs coding agents all day?

Just got laid off from a chill swe job. It was remote. F. What is the quickest way to get a chill remote job where I have responsibilities and autonomy?

5YOE in JS Fullstack/ Python. SF. On student visa.

First off, this thread is NOT a petition to rally against the moderation team. Considering the deluge of trash they deal with every day, I think they are doing a valiant job and are to be commended. Consider it merely a place to discuss, which is what HN does best.

That said, it's becoming more and more obvious every day that there is a tremendous amount of attempts by bots, and specifically AI agents, to inject slop into HN threads. I worry about the integrity of the discourse here and if the ever growing wave of garbage will overtake staff resources to deal with it. Is it time to implement captcha for HN? If so, should it be out of the box, or a new mechanism more tailored to the security and privacy-centric nature of the HN readership? Are captchas even still effective enough in the age of AI to warrant their use?

We’ve all seen the crazy “10 parallel agents” type setups, but I never saw it fitting my workflow.

What I usually do is I would have Claude Code build a plan, Codex find flaws in it, iterating until i get something that looks good. I’d give direction and make sure it follows my overall idea.

Implementation is working well on its own.

But this takes a lot of focus to get right for me, I can’t see myself doing it on the same project, multiple features.

Am I missing something?

I'm researching demand for an end-to-end encrypted social platform focused on private groups (family/friends) rather than public feeds.

Core thesis: "They don't own you" is becoming a movement. People want: - E2E encryption (platform can't read content) - True deletion (cryptographic, not just "marked as deleted") - Subscription model (no ads, no data mining) - Open source (verifiable claims)

The gap: Signal is for messaging, Discord isn't private, Facebook is surveillance capitalism. Nothing serves the "private social groups" use case with real privacy.

Before building, I need to validate if this is a real pain point or just something that sounds nice but nobody would actually switch for.

2-min anonymous survey: https://forms.gle/bfZYPfxMUBCc1iACA

Honest feedback appreciated—especially if the answer is "this wouldn't work because X."

The best phone in 2026 is a dying iPhone 12 mini. It’s the smallest smartphone still maintained by a major manufacturer (both software updates and repair parts), runs all apps securely & as designed, costs ~$150 with accessories available for dirt cheap (my favorite sleeve officially made by Apple [1] cost $129 on release but is now available for around $10). A dying battery makes the phone better - it forces you use the phone less and only when you really need it. Don’t replace your battery, just turn the phone off and you’ll feel immediately less stressed. If you don’t want to miss calls, get an old Apple Watch with cellular and use it as your main “phone”. Buy a dying iPhone mini to save the planet and yourself, and maybe if Apple sees enough active minis they will make another one...

[1]: https://apple.fandom.com/wiki/IPhone_12_Leather_Sleeve_with_MagSafe

Context for the uninitiated or younger HN audience:

Aaron Swartz was a programmer and an internet freedom activist. He co-authored the RSS spec at age 14, and helped build Reddit, Creative Commons, Markdown syntax, Open Library, and more. But he was much more than the sum of his outstanding code contributions. He believed public information hidden behind unreasonable paywalls should be free, and fought to make it actually public.

In 2008, he wrote a script to download millions of federal court documents from the government's paywalled PACER database. In due time, he was caught by federal authorities, but the case was closed without filing charges. Aaron had violated the terms of service, but he had not broken the law as the documents were public property. The cache is now permanently hosted on the Internet Archive.

In late 2010, he started running a script to download the JSTOR archive, a digital library that locks millions of academic journals, papers, books, and primary sources behind expensive paywalls. JSTOR caught on, and started firewalling him. Aaron bypassed the firewalls by entering an unlocked utility closet in the basement of MIT's Building 16 and connected his laptop to the network switch, hiding it under a cardboard box. This did not end well.

MIT and JSTOR found the laptop and contacted the authorities, who turned this into a federal sting operation, and used a camera to catch Aaron in the act. Federal prosecutors and U.S. attorneys tried to make an example out of him. Instead of a simple trespass or civil suit, they charged him with multiple felonies including CFAA and wire fraud, threatening him with 35 years in prison and $1 million in fines (comparable to sentences for manslaughter, bank robbery, and worse crimes). Aaron rejected a plea deal that would brand him a felon.

After three years, Aaron was still facing trial and the full weight of the federal government. On January 11, 2013, he took his life. He was 26 years old.

Today, we pay tribute to a pioneering builder and thinker of the open web.

Some links:

Aaron's manifesto on freeing academic knowledge: https://ia600101.us.archive.org/1/items/GuerillaOpenAccessManifesto/Goamjuly2008.pdf

Aaron's weblogs: https://github.com/joshleitzel/rawthought/tree/master

An excellent documentary on Aaron Swartz: https://archive.org/details/TheInternetsOwnBoyTheStoryOfAaronSwartz

Aaron’s keynote “How we stopped SOPA”: https://www.youtube.com/watch?v=Fgh2dFngFsg

16 year old Aaron speaking at the launch of Creative Commons: https://www.youtube.com/watch?v=YpT_V-DB1JU

Hey guys,

I've come all the way to SF to build a startup. But honestly doesn't feel like much has changed. Is there anything I need to do in order to leverage SF better? Or does being in SF doesn't really matter that much?

## My Implementation

I'm using a 64-bit layout:

- Bits 63-51: Quiet NaN signature (0x7FFC...)

- Bits 50-18: 32-bit payload (integers, string pool indices, etc.)

- Bits 17-3: Unused/ (15 bits)

- Bits 2-0: 3-bit type tag

So it allows me to have 5 tagged types: `TRUE_VAL`, `FALSE_VAL`, `STRING_VAL`, `CALLDATA_VAL`, `U32_VAL`

This is for a domain-specific VM I'm building for programmable task management (think "Vim for todo apps" - small core with scriptable behaviors). The VM is stack-based with:

- String pooling & instructions pooling (indices stored as NaN-boxed values) - Call stack for task instructions execution.

code is here : https://github.com/tracyspacy/spacydo/blob/main/src/values.rs

Is it just me or do you guys think Datadog pricing is crazy? I mean even to SEE APM you have to pay like $50/mo minimum. What if my app has just 10 requests per month, can I get for free? No, pay us. What if it's just me building? No, still pay us.

But hey we'll give you this metrics page for free which just basically tells you your RAM usage.

And the worst part is, there is no proper alternative for this. And people are ok with that too. Am I crazy for thinking this?

I’m researching patterns for edge / gateway telemetry where the network is unreliable (remote sites, industrial, fleets, etc.) and you need offline buffering + bounded disk + replay once connectivity returns.

Questions for folks running this in production:

What do you use today? (MQTT broker + ??, Kafka/Redpanda/NATS, Redis Streams, custom log files, embedded DB, etc.)

Where do you buffer during outages: append-only log, SQLite/RocksDB, queue-on-disk, something else?

How do you handle backpressure when disk is near full? (drop policy, compression, sampling, prioritization)

What’s your failure nightmare: corruption, replay storms, duplicates, “stuck” consumer offsets, disk-full, clock skew?

What guarantees do you actually need: zero-loss vs “best effort” (and where do you draw that line)?

What metrics/alerts matter most on gateways? (queue depth, replay rate, oldest event age, fsync latency, disk usage, etc.)

I’d love to learn what works, what breaks, and what you wish existing tools did better.

Dear Americans, please don’t take this the wrong way - I love the US, have friends there, and treasure memories I made there.

However, it seems plausible that the US is turning into a rogue, authoritarian, Russia-like state increasingly more friendly towards Russia and hostile towards Europe. I am a European who grew up in a country still occupied by Russia. I am increasingly more worried about building my projects on American platforms, using an American operating system, etc.

What if the US actually attacks Greenland or finds another way to be openly hostile to Europe? I am not saying it will happen. All I am saying is that it seems prudent to prepare. How would you do it?

It is currently impossible to unhook myself from the US, but I would like to minimize exposure.

I can’t do anything about things like building an alternative to VISA/MasterCard (except wait for the digital Euro), so I will focus on things I can actually do and ignore things like my government buying F-35s and possibly giving my health data to Palantir.

* Mobile phone - there are no real European alternatives; it’s just Apple vs Google. Samsung or HTC with Android seems like a less bad option.

* Operating system - I have been using Linux for ages, getting rid of Windows seems relatively easy.

* Social networks - I grew to hate them before the current US admin, never used TikTok or Instagram, and I mostly stopped using Facebook and Twitter around the time Musk bought Twitter.

* Stripe for payments - this will be hard, but I am experimenting with our local payment processor, and so far it seems surprisingly doable, but it is not a battle-tested solution like Stripe.

* Clerk authentication - doable, but a lot of work and worrying

* AWS - I had a surprisingly bad experience with AWS and switched to a local provider with a lot less functionality (that I mostly do not need) and a lot better support

* GitHub, Cloudflare... dear God, how could we Europeans allow ourselves to be that dependent on anyone? Everything I touch is American.

* Gmail - this will be hard (two decades of emails). Any advice?

* Anything AI-related - fuuu, I am lost here.

What am I missing/forgetting? What do/would you do in my place?

I really hope you will take this as a brainstorming exercise and not an attack on America. I really do love the US and hope its democracy turns out to be more resilient than it currently seems.

EDIT: Please kindly keep responses practical. Let’s not turn this into a political discussion. You might approach it as a “what if” exercise, even if you think what the current US admin is doing is great, Europeans deserve what they get, etc.

Looking for recommendations for sites similar to Downdetector