That’s a good takeaway. AWS is the ultimate Swiss army knife, but it is easy to misconfigure. Personally, when you are first learning AWS, I wouldn’t put more data in then you are willing to pay for on the most expensive tier. AWS also gives you options to set price alerts, so if you do start playing with it, spend the time to set cost alerts so you know when something is going awry.
Have a great day!
So you just asked the most confusing thing about AWS service names due to how names changed over time.
Before S3 had an archival tier, there existed a separate service that AWS named AWS Glacier Storage, and then renamed to AWS S3 Glacier.
Around 2012 AWS started adding tiers to S3 which made the standalone service redundant. I received you look at S3 proper unless you have something like a Synology that can directly integrate with the older job based API used by the original glacier service.
So, let’s say I have a 1TB archival file, single tarball, and I upload it to a brand new S3 bucket, without version, special features, etc, except it has a life cycle policy to move objects from S3 standard to S3 Glacier instant access after 0 days. So effectively, I upload the file and it moves to Glacier class storage.
The S3 standard is ~$24/tb/month, and lets say worst case scenario our data sits on standard for one whole day before moving.
$0.77+$0.005 (API cost of the put)
Then there is the lifecycle charge to move the data from standard to glacier, with one request per object each way. Since we only have one object the cost is
$0.004 out of standard
$0.02 into glacier
The cost of glacier instant tier is $4.1/tb/month. Since we would be there all but one day, the cost on the first bill would be:
$3.95
The second month onwards you would pay just the $4.1/month unless you are constantly adding or removing.
Let’s say six months later you download your 1tb archive file. That would incur a cost of up to $30.
Now I know that seems complicated and expensive. It is, because it is providing services to me in my former role as director of engineering, with complex needs and budgets to pay for stuff. It doesn’t make sense as a large-scale backup of personal data, unless you also want to leverage other AWS services, or you are truly just dumping the data away and will likely never need to retrieve it.
S3 is great for complying with HIPAA, feeding data into a cdn, and generally dumping data around in performant way. I’ve literally dropped a petabyte off data into S3 and it just took it and did its thing.
In my personal AWS account I use S3 as a place to dump cache contents built by lambda functions and served up by API gateway. Doing stuff like that is super cheap. I also use private git repos (code commit), private container registry (ecr), and container host (ECS), and it is nice have all of that stuff just click together.
For backing up my personal computer, I use iDrive personal and OneDrive, where I don’t have to worry about the cost per object, etc. iDrive (not an Apple service) let’s you backup multiple devices to their platform and keeps them versioned.
Anyway, happy to help answer questions. Have a great day.
Thanks for posting. I just deployed to my container host in AWS ECS and it’s working well in my testing. Very easy deployment with docker.
It’s complicated. I gave the most expensive pricing, which is their fastest tier and includes stripping across three availability zones and guarantees 11 nines of data durability. Additionally, the easy integration with all other AWS services and the feature richness of S3 buckets makes it hard to do a fair apple to apple comparison unless you really have well defined needs. So I gave the highest price to keep it simple, and for someone who says they just have a few GB, any cost should be trivial.
AWS S3 has a free tier that covers the first 5Gb. I recommend it because the AWS cli is excellent, and gives you lots of options for how to sync your data. The pricing is $0.023/GB/month after the free tier. It can be overwhelming to get into AWS but it is worth it to have access to the ultimate IT service swiss army knife.
I run a lot of tech, containerized workloads in AWS, home firewalls running on protectli boxes for all my family around the country, wireless controllers to run APs for my family around the country, but as I got older one thing I stopped rolling my own instance of was data backups. My data backs up to OneDrive and iDrive, so two copies of my data. My wife has access to both via shared credentials in a 1password folder that she knows how to access and uses regularly.
As I got older and I had a family, the pictures of our kids, wills, financial records, insurance documents are all just too important. Every service that holds my data is paid annually for less than $200/year total and auto renews. She could call either company and prove ownership if she ever did need help getting access. Also, I can easily share folders to her.
It’s funny how getting older makes you think of the sorts of issues enterprise teams have. Don’t implement solutions where you will be one deep, have a succession plan, and complexity is the enemy. All the tech I run now is fun and helpful, but can be replaced with a trip to BestBuy. The data and pictures however must be easy to retrieve for her.
So I don’t have a good self hosted solution for you other than to say that at some point it’s ok to change your strategy. And if you are worried about privacy, you can encrypt subsets of your data locally before it is backed up.
When writing basic business code, structuring the code well and having good naming standards means you shouldn’t need a ton of comments, but you should still have some. Plus, using structured function content blocks gives you intellisense in some languages and IDEs, which is important for code reuse in teams.
However, when I was doing scientific programming I’d have comments for almost every line at times where I put the mathematical formula and operations the line represents. Implementing a convolution neutral network with parameters to dynamically scale the layers or MPI stochastic simulations is much different than writing CRUD functions or basic business logic.
I got into computers at a young age in the early 90s. You couldn’t really do much without getting knowledgeable. I learned basic and then assembler to follow along with magazines that shipped game code for you to follow along with. I later went on to build my own 16 bit computer out of NAND gates, including ALU, wrote a rudimentary compiler, network stack, and OS, etc. Very primitive but functional. I really just wanted to figure out how it all worked through the full stack, and get my games working along the way.
I eventually learned more languages and launched a career in IT and moved through just about every role. Picked up a math degree along the way to help. Was a system programmer on an IBM zos mainframe using C, natural, and assembler. Was a.net developer for a while, an enterprise DBA, cloud and network engineer, and then eventually exited the technical career through management.
So I guess I just always was interested in how computers worked, and getting my games working. I left the technical roles one I felt I had figured out all that I really needed to and went on to other challenges. Still play games and tinker with my own projects though.
One thing I’ll add is I often found it helpful to glide them in which helps straighten the wires, then pull them out and trim the ends to be even. Then put back in connector, and make sure all pins touch all wire ends.
I agree. Unfortunately many folks who are attracted to security issues and topics don’t have a great holistic view of things. The idea of security is that something can go wrong and you are still ok, and that you apply context appropriate measures. Of course sending a password through email isn’t good, but it’s a gaming forum. A security conscious individual should have randomly generated passwords for everything and no reuse. Likewise, it wasn’t a bank or a security company, it was an old forum software for public discussions, so contextually this isn’t a top concern.
The cherry on top is that it appears to have been an old screenshot and already addressed.
That’s interesting. Any chance your ISP could have been qos’ing streaming video? Although Singapore would be about the one place where a VPN concentrator would help; it is pretty much the big fiber hub in that local region for East, West, North connectivity.
I’ve only ever used Oracle cloud in an enterprise environment, so I don’t know what features you have available. I’m also much more familiar with AWS. But you should be able to create a proxy endpoint in your present region, and traverse the cloud providers internal network. That would likely improve your streaming. You could also create a VPN endpoint in your current region and terminate your traffic inside your cloud providers network, but that would add protocol overhead.
I would look at tools like iperf to look at your packet loss because being further from your server will increase latency, but shouldn’t impact the streaming unless you also have packet loss.
I used to oversee WAN and peering operations for a large multi site. Residential ISPs almost never respond to reports of inefficient routes unless you are one of their peers, big business customers, or you really know your stuff and send in a detailed report showing asymetric routes, bad bgp info, etc.
As far as a VPN goes, that probably wouldn’t help either. You will probably increase the number of hops and latency. Your route will still egress your isp gateway, to your VPN provider, then travel over the Internet and to your remote server, while adding additional protocol overhead. Yes, it is remotely possible that there is an improved link from his regional VPN node to his remote provider, but unlikely from my experience with traffic engineering.
Here’s a great podcast from NPR about the expiration dates, and what means what in the world of best by labels.
Link: Best by sell by use by
https://www.npr.org/2022/07/15/1111850221/best-by-sell-by-use-by
I highly recommend Stephen Tetlock’s book, super forecasting, who is the sponsor of the project you mention.
One method of forecasting that he identified as effective was using a spreadsheet to record events that might occur over the next 6-18 months along with an initial probability based on good judgement and the factors you quoted. Then, every day look for new information that adjusts the forecast up or down by some, usually small percent. Repeat, and the goal is you will trend towards a reasonable %. I omitted many details but that was the jist.
Now, that’s for forecasting on a short ish timeframe. There is a place for more open ended reasoning and imagination, but you have to be careful not to fall prey to your own biases.
This particular forecast of OPs feels like it is ignoring several long running trends in technology adoption and user behavior without giving events that would address them, and forecasts something they care about doing better in the long-term, a source of bias to watch. I tend to agree with you that I think elements of this forecast are flawed.