000webhost

Web hosting

Friday, November 12, 2021

ABS Postcode Research Script, Random Stuff, and More

- I recently had to do some postcode/suburb based research for marketing and research purposes. Basically, I created a script to scrape ABS (and similar) data and then converted this into a CSV file. Sounds simple but is pretty tedious if you understand how the data is formatted. Please note, that you can't actually obtain this data easily via so called "data packs" from the ABS. If you're interested in this download the following script:
- description is as follows:
# This script helps enable postcode based research based on ABS
# data in Australia. Uncomment the relevant lines of code and
# and run the script in the proper sequence. This script assumes
# you know how to code.
#
# Note, that data processing can take a lot of time and that the
# final file is so large that it may crash a lot of modern
# spreadsheet programs. Most of the time, I query this final file
# using basic BASH code.
#
# As this is the very first version of the program it may be VERY buggy.
# Please test prior to deployment in a production environment.
#
# 1.00 - works to download data
# 1.01 - didn't realise that australian_postcodes.csv was incomplete.
#        got some incomplete fields which leads to data alignment issues.
# 1.02 - reparse_abs_curl_csv doesn't deal with fields/misalignment of data
#        buggy due to misaligned data. Switch from comma to tab for fields
#        due to internal parsing
# 1.03 - works on first 20 of 73*.txt range, works for header1, data1 extraction
#         downloads missing nt and act ranges (mainly curious about what was 
#         there. In reality, it didn't contain much data
# 1.04 - heaps of data alignment issues. This mainly has to do with an improper 
#          number of tabs that occur occasionally. Found a bug in header which 
#          involves / character seems to work completely now on all fields but can't 
#         check due to total column import size limitation in Libreoffice. Had a 
#         double dash bug that cropped up during creation of tsv files and printf
#         https://unix.stackexchange.com/questions/22764/dashes-in-printf
# 1.05 - code cleanup. Created function to offer choice in header/data being
#           collated
# 1.06 - more code cleanup. If I created a problem during cleanup that's your
#          problem! This is a FOSS script/program and you should have known 
#          what you were getting into!
- you may be interested in the following posts as well:
https://dtbnguyen.blogspot.com/2021/09/wikipedia-list-data-scan-pack-random.html
http://dtbnguyen.blogspot.com/2020/12/open-university-certificate-renaming.html
http://dtbnguyen.blogspot.com/2020/11/get-free-open-university-course-scripts_3.html
http://dtbnguyen.blogspot.com/2019/08/getting-free-education-and-training.html
https://dtbnguyen.blogspot.com/2020/01/it-business-startup-resources-random.html
http://dtbnguyen.blogspot.com/2020/10/cvresume-harvesting-research-notes.html
https://dtbnguyen.blogspot.com/2020/01/email-harvesting-notes-sperminator-ari.html
http://dtbnguyen.blogspot.com/2018/09/seek-email-crawler-random-stuff-and-more.html
http://dtbnguyen.blogspot.com/2018/08/create-usernames-security-script-random.html
https://dtbnguyen.blogspot.com/2019/08/finding-fundinginvestment-making-money.html
https://dtbnguyen.blogspot.com/2019/07/comparing-icos-random-stuff-and-more.html
https://dtbnguyen.blogspot.com/2019/07/online-marketing-and-sales-notes-random.html
https://dtbnguyen.blogspot.com/2019/02/cryptocurrency-market-manipulation.html
https://dtbnguyen.blogspot.com/2019/01/free-online-advertising-and-marketing.html
https://dtbnguyen.blogspot.com/2018/03/social-media-bot-coding-notes-random.html
http://dtbnguyen.blogspot.com/2018/01/social-media-comment-bot-random-stuff.html
https://dtbnguyen.blogspot.com/2018/03/compare-social-media-script-random.html
https://dtbnguyen.blogspot.com/2018/03/whitepaper-examine-script-random-stuff.html
https://dtbnguyen.blogspot.com/2018/03/random-picture-makingblending-script.html
https://dtbnguyen.blogspot.com/2018/02/email-address-harvesting-script-random.html
https://dtbnguyen.blogspot.com/2017/08/cryptocurrencies-cryptocurrency-mining.html
https://dtbnguyen.blogspot.com/2017/08/the-big-5-us-it-firms-arent-unbeatable.html
http://dtbnguyen.blogspot.com/2017/04/youtube-news-downloader-script-music.html
http://dtbnguyen.blogspot.com/2018/01/net-neutrality-or-googleyoutube.html
http://dtbnguyen.blogspot.com/2015/01/building-reaktor-synthesisers-download.html
http://dtbnguyen.blogspot.com/2017/04/linux-planet-blog-checker-script-github.html
http://dtbnguyen.blogspot.com.au/2017/03/prophetspre-cogsstargate-program-8.html
http://dtbnguyen.blogspot.com/2015/10/geo-politics-soundcloud.html
http://dtbnguyen.blogspot.com/2019/09/4coder-website-downloader-script-random.html
http://dtbnguyen.blogspot.com/2018/02/bypassing-ejay-downloader-manager.html
http://dtbnguyen.blogspot.com/2020/08/corbett-report-podcast-downloader.html

Random Stuff:
- as usual thanks to all of the individuals and groups who purchase and use my goods and services
http://sites.google.com/site/dtbnguyen/
http://dtbnguyen.blogspot.com.au/
- latest in science and technology
https://www.abc.net.au/news/2021-11-07/future-of-food-to-change-as-producers-aim-for-net-zero/100500904
https://www.dw.com/en/the-era-of-tech-whistleblowing-is-here-but-will-it-lead-to-lasting-change/a-59726084
https://itwire.com/security/solarwinds-investors-sue-company-over-supply-chain-attacks.html
https://arstechnica.com/science/2021/11/nasas-stalwart-mars-helicopter-is-back-and-better-than-ever/
- latest in finance and politics
https://techcrunch.com/2021/11/09/daniel-eks-fund-puts-e100m-into-defence-startup-helsing-ai-to-support-democracies/
https://www.news.com.au/finance/economy/world-economy/rba-reveals-concerns-for-australia-as-chinas-evergrande-fiasco-escalates/news-story/9352e7a2d8a0fffbe9178661350eb29c
https://www.9news.com.au/world/climate-change-usa-china-pledge-to-work-together-to-reduce-emissions/a5fc751e-a91c-4e63-8c6c-b50ec7286626?utm_source=newsshowcase&utm_medium=gnews&utm_campaign=CDAqEAgAKgcICjCks_MKMNHogwMw_J4n&utm_content=rundown
https://www.afr.com/world/north-america/warren-buffett-shows-his-fear-by-dumping-the-most-shares-since-2008-20211109-p59771?utm_medium=social&utm_campaign=nc&utm_source=Facebook
https://www.bodyandsoul.com.au/mind-body/why-sally-ragequit-her-job/news-story/9d7366a4c8678fd3886774c538806690?nk=9eecd8ca8cc6137cda8a7c22128d77e9-1636505017
https://www.abc.net.au/news/2021-11-08/nt-don-dale-class-action-settlement/100602268
https://www.smh.com.au/national/nsw/the-challenge-of-finding-a-final-resting-place-for-old-mattresses-20211106-p596jz.html
https://www.theguardian.com/film/2021/may/10/tom-cruise-nbc-golden-globes-hollywood-foreign-press-association
https://www.msn.com/en-au/news/australia/calls-for-more-testing-after-asbestos-found-in-imported-toys-and-building-materials/ar-AAQtZ63?ocid=BingNews
https://www.news.com.au/finance/money/wealth/bill-gates-could-have-ended-up-richer-than-elon-musk-and-jeff-bezos-combined/news-story/6888490c1108775158f81558a3729d7c
https://www.news.com.au/finance/money/costs/afterpay-moves-into-hospitality-with-australian-venue-co/news-story/b569dcf94efcde0e5eef2ba79852c24f
https://www.msn.com/en-au/news/australia/government-commits-to-expanding-electric-vehicle-charging-stations-but-no-subsidies-to-increase-uptake/ar-AAQrOhr?ocid=BingNews
https://www.theguardian.com/film/2021/may/11/its-not-just-racism-and-sexism-the-golden-globes-have-been-sunk-by-sheer-stupidit
https://www.abc.net.au/news/2020-08-27/what-is-brushing-why-is-it-on-rise-during-coronavirus-pandemic/12602084
https://www.theguardian.com/australia-news/commentisfree/2021/nov/06/writing-a-book-about-scott-morrison-the-fact-he-seemed-boring-wasnt-an-obstacle
https://www.rt.com/news/539643-state-street-denies-diversity-panels-reports/
https://www.rt.com/business/539627-musk-to-sell-tesla-stock-looming-tax-bill/
https://www.dw.com/en/twitter-users-push-elon-musk-to-sell-10-of-tesla-shares/a-59750343
https://www.msn.com/en-au/news/australia/gerry-harvey-taken-to-court-by-longtime-friend-s-widow-in-bid-to-stop-eviction/ar-AAQtsdS?ocid=BingNews
https://abcnews.go.com/International/wireStory/iran-bans-newspaper-linked-supreme-leader-poverty-81030399
https://www.theguardian.com/commentisfree/2021/nov/07/billion-dollar-race-ageing-planet-old-age
- latest in defense and intelligence
https://www.presstv.ir/Detail/2021/11/06/670011/Afghanistan-US-Taliban-Sohail-Kabul-airport-missing
https://www.abc.net.au/news/2021-11-10/us-veterans-say-burn-pits-iraq-afghanistan-now-killing-them/100594934
https://www.9news.com.au/world/china-satellite-images-show-mock-us-aircraft-carrier-desert-site/9cd5d727-5916-48dd-b10f-86b1d4744560
- latest in animal and environmental news
http://www.gioi.com.au/
https://www.mintienfriends.com/piggy-info
http://www.essentialbaby.com.au/news/how-buckshot-the-horse-upstaged-owners-maternity-shoot-surprised-she-didnt-have-that-baby-right-then-20210610-h1wf2h
https://cafemom.com/news/horse-photobombs-maternity-photos
- latest in music and entertainment
https://interestingengineering.com/a-pensioner-accidentally-ejected-himself-from-a-fighter-jet-mid-flight
https://www.msn.com/en-au/news/australia/indian-village-marks-end-of-diwali-with-massive-cow-poo-fight/ar-AAQpYRr?ocid=BingNews
https://www.theguardian.com/food/2021/nov/08/flash-in-the-pan-adam-liaws-20-minute-wok-dishes
https://www.abc.net.au/news/2021-11-08/big-bird-covid-vaccine-twitter-ted-cruz-sesame-street-education/100602662
https://doobydobap.com/recipe/hong-shao-rou#wpzoom-premium-recipe-card

Random Quotes:
- China has rattled Western observers for centuries, and the idea that it might be approaching a level of formidable heft is troubling to those who, condescendingly, see it as a naughty child who aspired to economic growth but could only do so as long as it behaved. In other words, they achieved success because we let them, and profited from our generosity. (The "we" here constitutes a good number of rapacious powers that stripped the country bare in the nineteenth century and turned the country into a giant opium addict).
...
In matters of gross domestic product, this could certainly count as an argument. With less than $10,000 per capita, the PRC comes in at 70th in the world stakes. In terms of amount, it was the equivalent of 15.6 percent of the US, 17 percent that of Australia. In a remark that seemed self-flagellating and stinging, the embassy admitted that Beijing still faced "the challenge of unbalanced development and underdevelopment."
The aping of US positions by satraps is a disease that has no obvious cure short of full, bold independence from Washington's clutching power. But as has been shown with conviction, Australia's commitment to the United States remains unquestionable and more than a touch sad. At least Tony Blair and his Britain were accused by their harshest critics of being substantive partners of a BDSM relationship with George W. Bush: Blair sporting the fishnets or perhaps being a subservient poodle. For Australian Prime Ministers, holding the drinks at an international reception in the US is sufficient to simply assume the views of the hosts. After all, the hosts can't be wrong, can they?
https://theaimn.com/scott-morrisons-china-thesis/
- My question isn't about the survivability of stealth fighters or its value. The technology is disruptive to an adversary's kill chain. Long range search radars can be countered with jamming, decoys, and a hard kill. Supposedly the IAF destroyed a JY-27 a few months ago in Syria. My question has to do with to what extent can a F-22/35 be detected by these radars. Detection will depend on the aircraft's profile and aspect but at what point does it show up on radar. I woulds assume that even though stealth fighters are optimized to defeat shorter wave length radars, it still has some ability to counter modern search radars.
http://www.f-16.net/forum/viewtopic.php?f=38&t=54642&p=420958#p420958
- Like Turkey, India has stood strong on its purchase of the S-400 but it is learnt that only partial payments have been made by New Delhi, given banking sanctions that are already in place for dealing with Russian defence entities. More stringent CAATSA (Countering America's Adversaries Through Sanctions Act) sanctions have also been threatened against nations purchasing the S-400 from Russia.
While there had been an impression that the US could give India CAATSA waivers for the S-400, recent statements by Washington suggest that this would not be the case.
The US stand has been that it will not allow its modern combat aircraft to be operated in an environment where the S-400 is also operational, as it would be able to map these aircraft, enabling software upgrades and modifications to improve performance.
In an effort to wean India away from the S-400, the US has already offered its NASAMS II (National Advanced Surface-to-Air Missile System) for protection of the national capital region against ballistic missiles. In addition, the US has also been in talks for its advanced Terminal High Altitude Area Defense (THAAD) and Patriot Advance Capability (PAC-3) defence systems with India, though these would come at a significantly steeper cost than the S-400 system.
https://economictimes.indiatimes.com/news/defence/us-may-offer-f-35-fighter-if-india-scraps-s-400-deal/articleshow/69698536.cms
- Lower procurement rates for the Lockheed Martin F-35 have reversed the programme's positive gains since 2015, raising overall development and procurement costs by almost 7% through Fiscal 2044.
After years of stabilizing and reducing total programme costs under Joint Programme Office executive officer Lt Gen Christopher Bogdan, lower planned procurement rates over the next 27 years drove overall, inflation-adjusted costs from $379 billion to $406.5 billion, according to a Selected Acquisition Report (SAR) summary released on 11 July.
The US Air Force decreased its maximum annual aircraft procurement rate from 80 F-35As per year to 60. The change extended planned purchases of F-35s by the Department of Defense by six more years from Fiscal Year 2038 to 2044.
The decreased production quantities and and six-year procurement phase extension not only led to higher programme costs, but an increase in unit recurring flyaway estimates over the programme's lifetime and an increase in average procurement unit cost and program acquisition unit costs, according to the JPO.
The JPO maintains that research, development, test and evaluation costs have remained steady and that negotiated near term unit recurring flyaway costs continue to decline.
At the start of the JSF programme, the June 2002 SAR estimated total acquisition costs at $226.4 billion, or $312.3 billion in current dollars.
The US Marine Corps, Navy and USAF had also planned to order 2,866 aircraft in 2002, but the navy later lowered its buy, reducing the total F-35 purchase to 2,443 aircraft today.
The overall cost estimate was released a day after the DOD awarded Lockheed a $4.49 billion undefinitised contract adjustment with a $5.57 billion ceiling for the 11th lot of low-rate initial production (LRIP). The deal is a placeholder while the JPO negotiates final LRIP lot 11 prices with Lockheed for 90 airframes and with Pratt & Whitney for F135 engines.
The JPO says the negotiations for Lot 11 should result in a lower price than finalised in LRIP lot 10, which resulted in a $94.6 million flyaway unit cost for the F-35A.
https://www.flightglobal.com/news/articles/lower-buys-raises-overall-f-35-acquisition-costs-by-439224/
- A. Gary Shilling is an American financial analyst and commentator who appears on a regular basis in publications such as Forbes magazine, The New York Times and The Wall Street Journal. He is President of A. Gary Shilling & Co., Inc.,[1] editor of A. Gary Shilling's Insight, and member of The Nihon Keizai Shimbun[2] Board of Economists. He is featured frequently on business shows on radio and television, and as a recognised orator, addresses conventions of global business groups like the Young Presidents' Organization. He was awarded a bachelor's degree in Physics from Amherst College, and a PhD in Economics from Stanford University. He has worked for Thornhill Securities, the Federal Reserve Bank of San Francisco, Merrill Lynch, and White Weld & Co., and Standard Oil Co NJ.[3]
In the spring of 1969, he was one of only a few analysts who correctly envisioned the recession at year's end,[4] and was almost a lone voice in 1973, when he forecast a monolithic international inventory-building fling, followed by the first significant recession since the Great Depression.
In the late 1970s, while most analysts presumed that waxing inflation would go on unabated, Shilling was the first to predict that America's infirm political climate would impede it.[5] He also foresaw various dangerous economic readjustment problems and a shift in investment strategy from a preference for tangible assets to an increased emphasis on stocks and bonds.
In June 2011, he predicted a 20% drop in housing in 2012 with a resulting global recession.[6] In October 2012 he predicted a global recession in 2013. [7]
In August 2015, he predicted that the price of oil "is headed for $10 to $20 per barrel" (it was $43/barrel at the time) due to higher productivity through fracking and OPEC not limiting production.[8]
https://en.wikipedia.org/wiki/Gary_Shilling
- The US believes that not only would the Russian air defence system be incompatible with NATO equivalents, but that it could possibly reveal weaknesses in the F-35 to Russia, rendering the jet's stealth capabilities useless.
The US has been trying to convince Turkey to ditch its deal with Russia ever since it was inked in 2017, but without success. Turkish President Recep Tayyip Erdogan has repeatedly stated that the deal will proceed as planned and that the S-400s will be delivered on schedule in July 2019. Erdogan has also suggested creating a working group to iron out US concerns regarding the S-400 systems, but the US has not agreed to participate.
https://sputniknews.com/middleeast/201906141075876549-germany-france-positions-turkey-purchase-russian-s-400-systems/
- Washington is really looking for three things from Beijing: play by the same set of rules everyone else is expected to play by, level the playing field with respect to a more equal trading relationship, and act as a responsible global leader that promotes global development and maintain peace. On its face, there is nothing unreasonable about any of this, but from the Chinese perspective, Beijing sees nothing wrong in doing things the Chinese way and cannot understand why any other nation might object.
Beijing wants Washington to mind its own business and not interfere in what it considers to be Chinese domestic issues. Moreover, China wants America to practice what it preaches and stop being hypocritical about its own rhetoric versus its own actions around the world. Additionally, Beijing, of course, wants Washington to stop trying to impede China's inevitable rise. There is clearly incongruity between what both are seeking and what either may be prepared to deliver. What differentiates this challenge from that of any other two nations is that both countries need each other in order to prosper and either of them is capable of causing insurmountable problems politically, economically, and militarily for the other.
https://nationalinterest.org/feature/america-needs-real-china-strategy-62567
- Chinese didn't sold anything to anyone expect J-7 in last couple of decades. No one bought J-10 nor J-11. Maybe they will sell JF-17 to someone. I don't count Pakistan they were partner.
And I really find funny when citation say Russians don't know to make RAM :D
When Russians visited our aircraft museum folks there ask them why they don't take parts of F-117 as Chinese did. Answer was simple "it is old tech compare to what we are working" so even though Russians didn't field stealth in past don't think they don't have clue about RAM, Tu-160 is probable first their plane where they apply RAM (some sources say it is Tu-22M3), intakes are threaded with graphite RAM and there are some sources (interview with constructor) which mentioned airframe have RAM coatings.
And if you don't like what Russians say you can googled CIA report from 1980s where CIA say lot more scientists and engineers works on RAM development in USSR then in USA.
http://www.f-16.net/forum/viewtopic.php?f=36&t=55693
- A senior commander of Iran's Islamic Revolution Guards Corps (IRGC) said late in May that the US presence in the Middle East is at its "weakest in history" of Washington's military presence in this region, amid spiraling tensions between the two countries.
"The Americans have been present in the region since 1833 and they are currently at the weakest state throughout the history of their presence in West Asia, and have the minimum number of warships in the Persian Gulf," Rear Admiral Ali Fadavi, the second-in-command of the IRGC said.
https://www.presstv.com/Detail/2019/06/17/598752/Shamkhani-cyber-espionage-CIA-Persian-Gulf

Market Consolidation/Neo-Feudalism, Random Stuff, and More

- it never occured to me until recently how consolidated things in the world were in the global market place. In this post we'll take a ...