Ask stories

I am interested in knowing the things the community is building specifically taking into account the NoGIL aspect of python. Like is someone building frameworks around using Threads instead of Async?

hi hn - i'm volunteering at Stanford next quarter to co-teach cs 153 (infrastructure at scale) - a course i wish had existed during my undergrad years. rather than pure theory, it's focused on how large-scale systems actually work in production

the format combines hands-on projects with a speaker series. we've confirmed some solid speakers (Jensen Huang from NVIDIA, Matthew Prince from Cloudflare etc), but i'm also keen to bring in perspectives from folks who don't fit the standard mold. tbh, many of the best systems eng/devs/infra ppl i've worked with are pretty weird - they think differently, take unconventional paths, and often learn by obsessively building and breaking things rather than following traditional routes. i think it would be cool for the students to realize its a feature, not a bug, to be weirdly obsessive

if you're interested in this kind of stuff, i'd value your thoughts on:

1/ who are the fascinating/unsung heroes in infra/systems eng that students should learn from? especially interested in people who've solved hard scaling problems through unconventional thinking or unique approaches

2/ what kind of projects do you think would fun and meaningfully demonstrate real-world infrastructure challenges while still being achievable in an academic quarter?

prerequisites are CS106/CS111 level programming. draft syllabus here: https://explorecourses.stanford.edu/search?view=catalog&filt...

email: anjney at alumni dot stanford edu if you prefer to share thoughts privately. thank you in advance for any and all help

LLMs are becoming a part of software engineering career.

The more I speak with fellow engineers, the more I hear that some of them are either using AI to help them code, or feed entire projects to AI and let the AI code, while they do code review and adjustments.

I didn't want to believe in it, but I think it's here. And even arguments like "feeding proprietary code" will be eventually solved by companies hosting their own isolated LLMs as they become better and hardware becomes more available.

My prediction is that junior to mid level software engineering will disappear mostly, while senior engineers will transition to be more of a guiding hand to LLMs output, until eventually LLMs will become so good, that senior people won't be needed any more.

So, fellow software engineers, how do you future-proof your career in light of, the inevitable, LLM take over?

--- EDIT ---

I want to clarify something, because there seems to be slight misunderstanding.

A lot of people have been talking about SWE being not only about code, and I agree with that. But it's also easier to sell this idea to a young person who is just starting in this career. And while I want this Ask HN to be helpful to young/fresh engineers as well, I'm more interested in getting help for myself, and many others who are in a similar position.

I have almost two decades of SWE experience. But despite that, I seem to have missed the party where they told us that "coding is not a means to an end", and realized it in the past few years. I bet there are people out there who are in a similar situations. How can we future-proof our career?

I have a project that I'm working on turning into a small business. I've done some part-time work (retainers and project-based) over the past year and it's gone well. It's relatively high pay for part-time work, leaving me time and flexibility to work on my own project.

The thing, I haven't really put much work into finding this kind of work. I've had a few opportunities land in my lap pretty nicely. Now, I need to seek out more work like this. I have ideas, but I'm curious to see how others are finding part-time work. Ideally, I would get 10-15/hr a week retainers, but project-based work is ok too. The key is that I can keep getting the work with consistency.

My corporate career was a cross between engineering and product management. I truly believe my best utility is the cross-over of the two. I'd be happy to do part-time leadership for small teams, take on independent projects, do things like build and maintain small apps/integrations, etc.

So:

1) How are you finding part-time work?

2) How do you sell yourself if you're more of a generalist like me?

Now that everybody and their mother are fuzzing in social media about LLM agents and agentic LLM systems (or something), are there actual examples of live applications that are based on an agentic LLM process flow?

I'd be curious to know and see such examples in order to derive some inspiration from them.

I'd love to use them, just they won't reply to any emails about their Live API. I can't seem to find any reasonable alternatives to https://www.browserling.com/api

I'm 3 years into my career as a software engineer. I became enthralled with programming while in graduate school (studying a completely different discipline) and decided to drop out to pursue a 2nd BS in computer science.

I work at a large F500 corporation that is attempting to brand itself as a tech company. But the reality is, there really isn't an engineering culture. I'm at an impasse with regard to my career and emphasizing my professional growth. I've identified what I would like to have moving forward, but I have no frame of reference to know whether it is realistic to find a balance to these key things:

1. *Passion for Craftsmanship* - Mentorship is a big ask of more experienced individuals. I'm eager to learn and to do so in an environment where people get excited building things. I recognize my development is largely my responsibility, but I'm desperate for feedback on how I can improve and grow.

2. *Available Opportunities* - I didn't make the career change for the money, but its nice to have. But not so nice, I'd want to be completely miserable for it. I've seen it be suggested that being in a major tech hub would be beneficial both in terms of salary as well as opportunity. At present I'm willing to relocate to wherever the opportunities exist. I moved across the US for my current role, but the surrounding region isn't bustling with SWE jobs should I find myself unemployed.

I don't dislike where I am at currently but I do dislike that I feel a sense of stagnation seeping in. My attempts at articulating my desire for a challenge at work get pushed aside and while my performance reviews keep coming in glowing, I feel stuck and don't feel like I am doing my best work.

How do you vet companies / jobs that have what your looking for in terms of growing your career?

Hey HN,

I've been leveraging LLMs to quickly build out React apps, which is great, but I'm really struggling with the design aspect. My apps function well, but they lack that polished look, and I end up in an unproductive loop of CSS adjustments.

Has anyone explored using LLMs specifically for design improvements? How can they be harnessed to create more visually appealing outcomes without the endless tweaking?

Thanks!

Hi HN - Senior SWE at FAANG who has basically stayed afloat via stimulants which gradually ruined my life and emotional well being.

I finally broke the chains and more or less replaced these bad habits with healthier ones, which has been great. However, I can barely function at work and my contributions have dropped to near zero. It’s causing me immense stress in a culture that is cutthroat.

How can I rebuild a normal way of working?

If your work brings you a sense of fulfillment, interest, or enjoyment I would love to hear more about it and what about your job allows you to feel this way.

I have realized I am someone who needs at least one of these things to be happy with work, and I’m currently not getting any of them. This next year I would like to seriously make an effort to get on track to change this for myself.

It's amazing how software from a major company that is less than 30 years old can get lost & forgotten so fast.

Building on a 2022 post about finding a copy of the long lost Slovenian OS/2 Warp 4 - https://www.os2museum.com/wp/slovenian-os-2-warp-4/ - some fellow software archaeologists have concluded some other localization editions are still missing - copied from the post's comments:

-- OBattler says: December 8, 2024 at 9:53 am It turns out Slovenian isn’t the only version missing of OS/2 Warp 4.0 – Portuguese (Portugal) and Portuguese (Brazil) are also missing, as are Dutch, French, Finnish, Norwegian, and Swedish, and a non-trial version of Hungarian.

OBattler says: December 8, 2024 at 10:01 am I just found FixPack 5 for OS/2 Warp 3.0 (or at least, references thereof) Arabic, French (Canada), Hebrew, Thai, Turkish, and, apparently, also Bulgarian and Lithuaian! So OS/2 Warp 4.0 must have also existed in these languages.

Source: https://ecsoft2.org/system-fixpacks-and-patches --

Anyone who can help find these CDs and help the os2museum.com preserve them (and optionally upload them to archive.org, too) is very welcome to join our cause.

Also looking for any (ex)IBM employees thst worked on the OS/2 localization projects and can help with this!

Marko Štamcar Computer History Museum Slovenia

Signal and Telegram are built around private, encrypted communication.

But every time, I join the meeting I see some random AI note-taker bot. It basically transcribes the most precise things ever - your meetings.

Why people stopped to care that their data - their voice, their meeting voice are processed on servers somewhere?

Hi folks,

I am working on a game preservation project, where the data set holds 10.4 TiB. It contains 1044 earlier versions of a single game in a multitude of different languages, architectures and stages of development. As you can guess, that means extreme redundancy.

The goals are: - bring the size down - retain good read speed (for further processing/reversing) - easy sharable format - lower end machines can use it

My choice fell on the BTRFS filesystem, since it provides advanced features for deduplication, which is not as resource hungry as ZFS. Once the data is processed, it no longer requires a lot of system resources.

In the first round of deduplication, I used "jdupes -rQL" (yes, I know what -Q does) to replace exact copies of files in different directories via hardlinks to minimize data and metadata. This got it down to roughly 874 GiB already, out of which 866 GiB are MPQ files. That's 99,08%... everything besides is a drop in the bucket.

For those uninitiated: this is an archive format. Representing it as a pseudo-code struct it looks something like this { header, files[], hash_table[], block_table[] } Compression exists, but it is applied to each file individually. This means the same file is compressed the same way in different MPQ archives, no matter the offset it happens to be in.

What is throwing a wrench into my plans of further data deduplication are the following points: - the order of files seems not to be deterministic when MPQ files were created (at least I picked that up somewhere) - altered order of elements (files added or removed at the start) causes shifts in file offsets

I thought for quite some time about this, and I think the smartest way forward is, that I manually hack apart the file into multiple extents at specific offsets. Thus the file would contain of an extent for: - the header - each file individually - the hash table - the block table It will increase the size for each file of course, because of wasted space at the end of the last block in each extent. But it allows for sharing whole extents between different archives (and extracted files of it), as long as the file within is content-wise the same, no matter the exact offset. The second round of deduplication will then be whole extents via duperemove, which should cut down the size dramatically once more.

This is where I am hanging right now: I don't know how to pull it off on a technical level. I already was crawling through documentation, googling, asking ChatGPT and fighting it's hallucinations, but so far I wasn't very successful in finding leads (probably need to perform some ioctl calls).

From what I imagine, there are probably two ways to do this: - rewrite the file with a new name in the intended extent layout, delete the original and rename the new one to take it's place - rewrite the extent layout of an already existing file, without bending over backwards like described above

I need is a reliable way to, without chances of the filesystem optimizing away my intended layout, while I write it. The best case scenario for a solution would be a call, which takes a file/inode and a list of offsets, and then reorganizes it into that extents. If something like this does not exist, neither through btrfs-progs, nor other third party applications, I would be up for writing a generic utility like described above. It would enable me to solve my problem, and others to write their own custom dedicated deduplicaton software for their specific scenario.

If YOU - can guide me into the right direction - give me hints how to solve this - tell me about the right btrfs communities where I can talk about it - brainstorm ideas I would be eternally grateful :) This is not a call for YOU to solve my problem, but for some guidance, so I can do it on my own.

I think that BTRFS is superb for deduplicated archives, and it can really shine, if you can give it a helping hand.

I'm thinking of purchasing the Devin subscription, so I can give it screenshots of a web-app, and have it just implement the flows. But ... I don't know if it will be capable of doing so.

Has anyone used Devin for this type of thing?

Apache Kafka is critical for real-time streaming, but its costs, especially with Confluent, can be prohibitive.

AutoMQ offers a fresh take: a cloud-native Kafka implementation that reduces costs by rethinking traditional architectures.

Preconditions for Comparison

Peak Throughput: 0.1 GB/s

Average Throughput: 0.01 GB/s

Monthly Data Transfer: 25,920 GB

Storage Volume: 2,592 GB

Architecture: Multi-AZ (AWS)

Data Retention: 3 Days

Key Cost Differences:

Confluent Total: $17,369/month

AutoMQ Total: $718/month (95.8% less) How?

Stateless Brokers: No shared-nothing overhead.

Shared Storage: Amazon S3 eliminates cross-AZ replication costs.

Elastic Scaling: Decouples compute and storage for efficiency.

Core Benefits: Massive cost savings: Compute (-94.7%), Network (-99%), Storage (-78.3%).

Simpler operations with cloud-native scaling.

Retains Kafka's core performance without heavy infrastructure.

For anyone working with Kafka: Is this a sustainable solution for cost reduction?

Would you consider migrating to a cloud-native Kafka setup like AutoMQ?

A few months back I was working on a project to help founders that sell to SMBs get better quality leads (Current solutions like Zoominfo and Apollo don’t do very well for the SMB market). Of course, I wanted to do this as quickly as possible with as little code as possible. We found that people were manually going through Google Maps to find SMBs. They would use the search and manually type in the businesses they were looking for. For example, they would type “restaurants” and manually call/email them. What we decided to do was gather the Google Maps data autonomously and surface that to our customers so they could take all of it. The problem was that we would need a bunch of data from Google Maps to pull it off. We would need to grab all the SMBs across the United States which is a huge undertaking. Initially, I tried no-code AI web scraping solutions and they worked horribly. For some reason, I couldn’t even get them to scroll down on the page. I was also able to reverse engineer their open-source code and discover that they were taking the entire web page and passing it into GPT to extract data. That just burned my Openai bill. I then tried the semi-code approach where I would use something like Apify or Google Places API to scrape the businesses. This worked better but still, there was an issue of price at the scale we wanted. Eventually, we ended up writing our scraper for the task. This experience was so horrible we ended up creating https://potarix.com . Our approach is fundamentally different because we’re focused on generating web scraping code in the backend (Think bolt.new but for data extraction). Right now the app is pretty bare bones, simply type in a url and prompt it with the data you want from the site and you’ll get it. However, we have some sauce we built in house for our custom clients that we’ll be releasing in the SAAS soon. We’ve built functionality to click, type, scroll, etc. on the page. We’ll also be providing our generated script soon as well, so you guys can tweak the code accordingly to bring the script to 100% accuracy!

Hi HN,

I’m building a job aggregator with a live data platform that provides in-depth market analysis. I’m currently focused on improving how I extract skills from job postings. While my current extraction setup achieves ~90% accuracy, it struggles with edge cases and lacks flexibility, particularly when skills are phrased in unexpected ways.

1.The Problem: 1.1: Lack of flexibility: The system only captures predefined phrases. If a job post says something like "proficiency in spreadsheets" or "experience with advanced reporting tools", it misses that Excel is likely required.

1.2: Manual maintenance: Constantly updating JSON files to account for new variations is tedious and unsustainable as the project grows.

2.Current Setup: 2.1: Keyword-based extraction: I maintain a JSON file with predefined skill variations. Example:

    "programming_languages": {
        "JavaScript": ["javascript", "js" ...],
         ...

3. Constraints: 3.1: Lightweight: I’m avoiding heavy ML models or resource-intensive pipelines to keep server costs low.

3.2: Flexible: I need a solution that better handles synonyms, context, and unexpected phrasing with minimal manual input.

3.3: Free or open-source: Ideally, something I can plug into my existing server setup without added costs.

4. My Questions: 4.1: How can I improve this process to make it more robust and context-aware?

4.2:Are there lightweight tools, heuristics, or libraries you’d recommend for handling variations and semantic similarity?

4.3: Would pre-trained embeddings (e.g., GloVe, FastText) or other lightweight NLP methods help here?

I’d love to hear from anyone who’s tackled similar challenges in NLP or information extraction. Any suggestions on balancing accuracy, flexibility, and computational efficiency would be greatly appreciated!

If anyone is interested in what my current market analysis looks like, I am leaving a link for you to analyze https://careercode.it/market

It seems like the longer I use the Internet the more I'm aware of just how many people use it exclusively to act like ass holes without recourse.

Particularly insidious are people who seem to poison any positive outcomes by spreading falsehoods and doubt. Sometimes even simply by having a shitty attitude.

My question to HN is: how do you deal with people who seem to go out their way to spread negative comments.

We're literally customers (ads) of facebook. No warning. Instagram claims we "violated community standards" on our business account, where we have only been posting our very above-board podcast videos.

I've already submitted a request for appeal, which was ~immediately denied. My FB account of 15+ years, plus all my groups, my various pages, and my followers, my business assets - all gone?

It's hard to even know what happened to trigger this. We've been making maybe 1-2 posts per day at most on the instagram account, and maybe have like 15 followers. Obviously the loss of the personal account + personal groups has a much bigger impact to me personally... but it's also crazy that they would clobber an active small business, that's paying them no less.

Can anyone at Meta please help?

Hey guys

I’m having a hard time keeping up with expectations at my job. I’ve been at a company for 3 years and last year we had a new manager who started keeping metrics.

Some of the metrics include how fast turn around times are and comments on MRs.

Our tickets are expected to be done in 3 days and we don’t do sizing or anything. We manage about 10 services not including the infrastructure which we also manage.

I also get a lot of comments on my MRs about minor issues like spacing or formatting which only my tech lead cares about. Unfortunately my manager considered this as a bad thing.

We do everything from infrastructure to front end. This year my fiancé has been sick and I’ve kind of been unable to keep up with expectations. I’m considered a low performer along with 3/4ths of my team. We’re a 4 man engineering team.

It seems like I can’t reasonably meet expectations but the job market is so bad I don’t think I’d survive.

I have a small software company I built for fun with 4,000 users but it’s not making money for me to live on.

Is it even possible to get a job without a referral in this market? I plan on riding it out until I get fired or the market gets better (it won’t). Are these expectations unrealistic?

Shameless plug for my comic collection app:

https://apps.apple.com/us/app/tabi-comic-collection/id6476126980

Been debugging an issue with a colleague for a few days now and we are coming up empty handed. Most of our production app relies on gevent. The application works on macOS 12.x, 13.x and on all flavors of Linux ... but as of now 15.x is causing some very hard to debug issues. I get a sense this has something to do with the way gevent is patching things like `socket`. Instead of seeing immediate feedback like a failure to connect to an external resource, the app just silently hangs/fails.

I am reaching out here as a last ditch effort to try and see if anyone else in the community has experienced issues specifically in regards to the combo of macos 15 and gevent.

In his recent talk Ilya S. said that the data running out is a fundamental constraint on the scaling laws. He said "we have but one internet".

But I don't understand: there is so much data in the real world beyond the internet. Webcams. Microphones. Cars. Robots... Everything can collect multimodal data and more importantly (for robots) even get feedback loops from reality.

So isn't data functionally infinite? And the only thing standing in the way is the number of sensors and open datastreams and datasets.

Please help me understand

Hi HN,

I recently started a new position as a "program analyst" in the context of software for public administration in EU. There are two teams involved in the project, team A who are basically like product owners and architects, doing the conception and deciding of the priorities, and team B, the devs, who actually make the software.

My job in team B is to act as an interface between team A and the devs, I'm supposed to have a good overview of the project, keep track of the progress made in the code (bugs, new features etc.) so I can update team A on what's going on and in the end synchronize the teams. I also have to manage our git repo. It was clear from the beginning that I am not supposed to code. Overall I don't feel super enthusiastic about the project itself but I must admit that for a junior the working conditions are really good (working hours, holidays and salary) and the team is nice.

The contract is set for 3 years, since I'm a junior with low experience I fear that at the end of the contract I will lack some coding experience and have trouble getting a new job. So I was wondering if there are similar positions in the private sectors? And basically, how can I get the most out of this job?

Thanks!

Hello,

I am a developer with few skills in DevOps and I am looking for a solution to run small APIs on docker containers. My goal is to have the least amount of infrastructure to manage as I am short on time and I am relatively new to it so I see the "severless" abstraction a good thing for my current situation/

I think that Lambda is cool for that but I found the AWS UI quite overwhelming. I checked Digital Ocean functions but you can't use a container. Do you know other similar products ?

I recently came across insights into the growing bioherbicides market and found some fascinating trends worth discussing with the community. The market is projected to grow from $2.22 billion in 2024 to $5.86 billion by 2031, fueled by a shift toward sustainable agriculture, organic farming, and weed resistance to chemical herbicides.

What’s particularly intriguing is how regulations are playing a pivotal role. For example:

The EU's Farm to Fork strategy targets a 50% reduction in chemical pesticides by 2030. Canada has allocated CAD 75 million for sustainable agriculture research, focusing on biopesticides. Innovation is also heating up. One example is Marrone Bio Innovations' upcoming bioherbicide (launching 2025), which targets herbicide-resistant weeds using Streptomyces acidiscabies.

This raises some interesting questions:

Are bioherbicides the solution to making agriculture more sustainable, or are there hidden challenges (e.g., scalability, cost, or efficacy)? How critical are regulatory changes in driving the adoption of such technologies? Is there potential for open innovation or collaboration between startups and larger agricultural companies to speed up progress in this space?

There isn't any official page for this that I can find, but I thought HN might enjoy the 2024 Lichess recap, it's a really nicely done recap of the year on Lichess. I'll post a link to a video I found in a search and a couple forum posts in the comments.

A similar discussion from 2021 that didn't track a lot of attention: https://news.ycombinator.com/item?id=29165586

And the only answer from eesmith cites a very old implementation, I'm wondering what's being used nowadays in the HN code base

The new design update to the AWS UI is incredibly bad in my opinion. Literally every button is blue now.

I'm lucky to have 2 offers in this market as a New Grad, however they are vastly different and both have their pros and cons so I'm having trouble deciding.

They're both in MCOL suburb areas and are decent names within their own industry, but they're not super popular or tech companies. One is notably higher pay ($110k USD vs $90k) but a lot worse WLB (in person 5 days, 45-50hrs a week, almost no vacation/sick days, notably higher rate of turnover) vs the other co. is hybrid, ~30-35 hours/week, slower moving, and a lot more vacation/sick days overall. Both have decent benefits.

The tech stacks for both companies vary a bit but they're both on the legacy side of things with a lot of work done maintaining old crappy code, also it's really team dependent (I don't know what team I will be on). The caveat for the second lower-paying company is that I will have the chance to try out two different teams for 6 months to see which I like.

My goal is either FAANG or a major tech company within a couple years. I would love the extra money but I also value WLB quite a bit so its either between putting my head down and grinding or trying to enjoy life after uni. The learning opportunities at either company seem to be completely variable depending on which team I get put on.