annoyances

[top]

software

[top]

macport and fink

[top]

hibernate

[top]

presentations

[top]

mount iso

[top]

using disk utility 
or 
hdiutil mount sample.iso

create iso compatible on windows

[top]

* use disk utility to create the CDR file
* then, enter this line of code to transform the .cdr to an ISO file:
hdiutil makehybrid -iso -joliet -o [filename].iso [filename].cdr

hdiutil makehybrid -iso -joliet -o tmp.iso tmp -ov

burn DVD

[top]

burn ISO linux installer

[top]

migrate boot device to SSD

[top]

programming editors

[top]

install fonts

[top]

juniper vpn

[top]

compare folders

[top]

SecureCrt

[top]

1) Install SecureCRT on windows and point it to C:\Dropbox\Putty\SecureCRT\Config location
2) Install SecureCRT on mac and copy all Sessions file to windows

Karl-MacBook:Sessions karl$ pwd
/Users/karl/Library/Application Support/VanDyke/SecureCRT/Config/Sessions

Karl-MacBook:Sessions karl$ cp -rpv * /Users/karl/Dropbox/Putty/SecureCRT/Config/Sessions/
Default.ini -> /Users/karl/Dropbox/Putty/SecureCRT/Config/Sessions/Default.ini
__FolderData__.ini -> /Users/karl/Dropbox/Putty/SecureCRT/Config/Sessions/__FolderData__.ini
v2 -> /Users/karl/Dropbox/Putty/SecureCRT/Config/Sessions/v2
v2/__FolderData__.ini -> /Users/karl/Dropbox/Putty/SecureCRT/Config/Sessions/v2/__FolderData__.ini
v2/enkdb01.ini -> /Users/karl/Dropbox/Putty/SecureCRT/Config/Sessions/v2/enkdb01.ini

3) Create symbolic link on Sessions folder to Dropbox

cd /Users/karl/Library/Application Support/VanDyke/SecureCRT/Config/
rm -rf Sessions/
ln -s /Users/karl/Dropbox/Putty/SecureCRT/Config/Sessions Sessions

outlook on vbox with zimbra connector

[top]

safeboot, rescue mode

[top]

format hard disk for time machine - MAC and NTFS

[top]

enable SSH to localhost

[top]

SSD migration

[top]

restart audio/sound service without reboot

[top]

sudo kill -9 `ps ax|grep 'coreaudio[a-z]' | awk '{print $1}'`
sudo kextunload /System/Library/Extensions/AppleHDA.kext 
sudo kextload /System/Library/Extensions/AppleHDA.kext

command tab doesn't work, dock not responding

[top]

killall -9 Dock

killstuff

[top]

Karl-MacBook:~ root# cat killstuff.sh 
kill -9 `ps -ef | grep -i "macos/iphoto" | grep -v grep | awk '{print $2}'`
kill -9 `ps -ef | grep -i "macos/itunes" | grep -v grep | awk '{print $2}'`
kill -9 `ps -ef | grep -i "GoogleSoftwareUpdate" | grep -v grep | awk '{print $2}'`

vnc, screensharing

[top]

vnc://192.168.1.9

verify, repair hard disk

[top]

dot_clean ._ underscore files

[top]

get CPU information

[top]

-- 15 inch 
AMAC02P37MYG3QC:~ kristofferson.a.arao$ system_profiler SPHardwareDataType
Hardware:

    Hardware Overview:

      Model Name: MacBook Pro
      Model Identifier: MacBookPro11,2
      Processor Name: Intel Core i7
      Processor Speed: 2.2 GHz
      Number of Processors: 1
      Total Number of Cores: 4
      L2 Cache (per Core): 256 KB
      L3 Cache: 6 MB
      Memory: 16 GB
      Boot ROM Version: MBP112.0138.B16
      SMC Version (system): 2.18f15
      Serial Number (system): C02P37MYG3QC
      Hardware UUID: F0DCC410-9E8A-5D77-98E3-C7767EB0CF8F


-- 13 inch 
Karl-MacBook:~ karl$ system_profiler SPHardwareDataType
Hardware:

    Hardware Overview:

      Model Name: MacBook Pro
      Model Identifier: MacBookPro9,2
      Processor Name: Intel Core i7
      Processor Speed: 2.9 GHz
      Number of Processors: 1
      Total Number of Cores: 2
      L2 Cache (per Core): 256 KB
      L3 Cache: 4 MB
      Memory: 16 GB
      Boot ROM Version: MBP91.00D3.B0C
      SMC Version (system): 2.2f44
      Serial Number (system): C1MK911MDV31
      Hardware UUID: 78B26406-BF78-531D-BCFB-1C3289BD44A5
      Sudden Motion Sensor:
          State: Enabled

sysctl -n machdep.cpu.brand_string

Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz

sysctl -a | grep machdep.cpu

machdep.cpu.max_basic: 13
machdep.cpu.max_ext: 2147483656
machdep.cpu.vendor: GenuineIntel
machdep.cpu.brand_string: Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz
machdep.cpu.family: 6
machdep.cpu.model: 70
machdep.cpu.extmodel: 4
machdep.cpu.extfamily: 0
machdep.cpu.stepping: 1
machdep.cpu.feature_bits: 9221959987971750911
machdep.cpu.leaf7_feature_bits: 10155
machdep.cpu.extfeature_bits: 142473169152
machdep.cpu.signature: 263777
machdep.cpu.brand: 0
machdep.cpu.features: FPU VME DE PSE TSC MSR PAE MCE CX8 APIC SEP MTRR PGE MCA CMOV PAT PSE36 CLFSH DS ACPI MMX FXSR SSE SSE2 SS HTT TM PBE SSE3 PCLMULQDQ DTES64 MON DSCPL VMX EST TM2 SSSE3 FMA CX16 TPR PDCM SSE4.1 SSE4.2 x2APIC MOVBE POPCNT AES PCID XSAVE OSXSAVE SEGLIM64 TSCTMR AVX1.0 RDRAND F16C
machdep.cpu.leaf7_features: SMEP ERMS RDWRFSGS TSC_THREAD_OFFSET BMI1 AVX2 BMI2 INVPCID FPU_CSDS
machdep.cpu.extfeatures: SYSCALL XD 1GBPAGE EM64T LAHF LZCNT RDTSCP TSCI
machdep.cpu.logical_per_package: 16
machdep.cpu.cores_per_package: 8
machdep.cpu.microcode_version: 15
machdep.cpu.processor_flag: 5
machdep.cpu.mwait.linesize_min: 64
machdep.cpu.mwait.linesize_max: 64
machdep.cpu.mwait.extensions: 3
machdep.cpu.mwait.sub_Cstates: 270624
machdep.cpu.thermal.sensor: 1
machdep.cpu.thermal.dynamic_acceleration: 1
machdep.cpu.thermal.invariant_APIC_timer: 1
machdep.cpu.thermal.thresholds: 2
machdep.cpu.thermal.ACNT_MCNT: 1
machdep.cpu.thermal.core_power_limits: 1
machdep.cpu.thermal.fine_grain_clock_mod: 1
machdep.cpu.thermal.package_thermal_intr: 1
machdep.cpu.thermal.hardware_feedback: 0
machdep.cpu.thermal.energy_policy: 1
machdep.cpu.xsave.extended_state: 7 832 832 0
machdep.cpu.xsave.extended_state1: 1 0 0 0
machdep.cpu.arch_perf.version: 3
machdep.cpu.arch_perf.number: 4
machdep.cpu.arch_perf.width: 48
machdep.cpu.arch_perf.events_number: 7
machdep.cpu.arch_perf.events: 0
machdep.cpu.arch_perf.fixed_number: 3
machdep.cpu.arch_perf.fixed_width: 48
machdep.cpu.cache.linesize: 64
machdep.cpu.cache.L2_associativity: 8
machdep.cpu.cache.size: 256
machdep.cpu.tlb.inst.large: 8
machdep.cpu.tlb.data.small: 64
machdep.cpu.tlb.data.small_level1: 64
machdep.cpu.tlb.shared: 1024
machdep.cpu.address_bits.physical: 39
machdep.cpu.address_bits.virtual: 48
machdep.cpu.core_count: 4
machdep.cpu.thread_count: 8
machdep.cpu.tsc_ccc.numerator: 0
machdep.cpu.tsc_ccc.denominator: 0

sysctl -a | grep machdep.cpu | grep core_count
machdep.cpu.core_count: 4

sysctl -a | grep machdep.cpu | grep thread_count
machdep.cpu.thread_count: 8

get memory information

[top]

hostinfo | grep memory

850 EVO mSATA vs 2.5

[top]

Caffeine - keep your mac awake

[top]

teamviewer reset id

[top]

install gnu parallel

[top]

pdftotext

[top]

._ in dropbox SSD

[top]

ms word blank images

[top]

dropbox conflicted copy

[top]

macbook recovery

[top]

closed lid no sleep

[top]

disk encryption

[top]

diskutil cs list 
diskutil list
diskutil cs unlockVolume 110B820A-52A5-4E3D-AB1C-FCF0263DC5A6 
diskutil mount /dev/disk3

diskutil list | grep "vm" | awk '{print $NF}' | sed -e 's/s[0-9].*$//'
diskutil eject disk4

disk unlock encrypted

[top]

Spinning Down Unmounted Disks in OS X

[top]

ulimit - too many open files

[top]

seems like there is an entirely different method for changing the open files limit for each version of OS X!

For OS X Sierra (10.12.X) you need to:

1. In Library/LaunchDaemons create a file named limit.maxfiles.plist and paste the following in (feel free to change the two numbers (which are the soft and hard limits, respectively):

<?xml version="1.0" encoding="UTF-8"?>  <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"  
        "http://www.apple.com/DTDs/PropertyList-1.0.dtd"><plist version="1.0">  <dict><key>Label</key><string>limit.maxfiles</string><key>ProgramArguments</key><array><string>launchctl</string><string>limit</string><string>maxfiles</string><string>64000</string><string>524288</string></array><key>RunAtLoad</key><true/><key>ServiceIPC</key><false/></dict></plist> 
2. Change the owner of your new file:

sudo chown root:wheel /Library/LaunchDaemons/limit.maxfiles.plist
3. Load these new settings:

sudo launchctl load -w /Library/LaunchDaemons/limit.maxfiles.plist
4. Finally, check that the limits are correct:

launchctl limit maxfiles

macos sierra bugs

[top]

Not Allowing Identified Developer App Downloads

[top]

sudo spctl --master-disable

OS X kernel asynchronous I/O limits

[top]

sudo sysctl -w  kern.aiomax=10240 kern.aioprocmax=10240 kern.aiothreads=16
sudo spctl --master-disable

Remove Symantec software for Mac OS using RemoveSymantecMacFiles

[top]

read mac crash reports

[top]

mds_Stores some times used 100% of CPU

[top]

how to run a script in mac on startup

[top]

uninstall spotify

[top]

disable internal keyboard

[top]

end

[top]

javascript

[top]

python

[top]

R

[top]

c

[top]

• Text Processing in Python https://www.amazon.com/Text-Processing-Python-David-Mertz/dp/0321112547/ref=sr_1_fkmr2_3?ie=UTF8&qid=1486829301&sr=8-3-fkmr2&keywords=python+text+processing+tricks
• Python 3 Text Processing with NLTK 3 Cookbook https://www.amazon.com/Python-Text-Processing-NLTK-Cookbook/dp/1782167854/ref=sr_1_fkmr2_1?ie=UTF8&qid=1486829301&sr=8-1-fkmr2&keywords=python+text+processing+tricks
• Text Analytics with Python: A Practical Real-World Approach to Gaining Actionable Insights from your Data https://www.amazon.com/Text-Analytics-Python-Real-World-Actionable/dp/148422387X/ref=pd_rhf_se_s_cp_1?_encoding=UTF8&pd_rd_i=148422387X&pd_rd_r=BQ961BNXXCRJC8FF1M0T&pd_rd_w=nXtOu&pd_rd_wg=f3MpB&psc=1&refRID=BQ961BNXXCRJC8FF1M0T
• Automate the Boring Stuff with Python: Practical Programming for Total Beginners https://www.amazon.com/Automate-Boring-Stuff-Python-Programming/dp/1593275994/ref=pd_rhf_se_s_cp_2?_encoding=UTF8&pd_rd_i=1593275994&pd_rd_r=BQ961BNXXCRJC8FF1M0T&pd_rd_w=nXtOu&pd_rd_wg=f3MpB&psc=1&refRID=BQ961BNXXCRJC8FF1M0T
• Violent Python: A Cookbook for Hackers, Forensic Analysts, Penetration Testers and Security Engineers https://www.amazon.com/Violent-Python-Cookbook-Penetration-Engineers/dp/1597499579/ref=pd_rhf_se_s_cp_6?_encoding=UTF8&pd_rd_i=1597499579&pd_rd_r=BQ961BNXXCRJC8FF1M0T&pd_rd_w=nXtOu&pd_rd_wg=f3MpB&psc=1&refRID=BQ961BNXXCRJC8FF1M0T
• Data Wrangling with Python: Tips and Tools to Make Your Life Easier https://www.amazon.com/Data-Wrangling-Python-Tools-Easier/dp/1491948817/ref=pd_rhf_se_s_cp_5?_encoding=UTF8&pd_rd_i=1491948817&pd_rd_r=BQ961BNXXCRJC8FF1M0T&pd_rd_w=nXtOu&pd_rd_wg=f3MpB&psc=1&refRID=BQ961BNXXCRJC8FF1M0T
• Text Mining with R: A tidy approach https://www.amazon.com/Text-Mining-R-tidy-approach/dp/1491981652/ref=sr_1_1?ie=UTF8&qid=1486829376&sr=8-1&keywords=R+text+processing
• Mastering Text Mining with R https://www.amazon.com/Mastering-Text-Mining-Ashish-Kumar/dp/178355181X/ref=pd_sbs_14_1?_encoding=UTF8&pd_rd_i=178355181X&pd_rd_r=QJR23QWXME6RM0EX70S9&pd_rd_w=1qsnW&pd_rd_wg=Qb4hQ&psc=1&refRID=QJR23QWXME6RM0EX70S9
• An Introduction to Information Theory: Symbols, Signals and Noise (Dover Books on Mathematics) https://www.amazon.com/Introduction-Information-Theory-Symbols-Mathematics/dp/0486240614/ref=sr_1_8?ie=UTF8&qid=1486829376&sr=8-8&keywords=R+text+processing
• Automated Data Collection with R: A Practical Guide to Web Scraping and Text Mining https://www.amazon.com/Automated-Data-Collection-Practical-Scraping-ebook/dp/B014T25K5O/ref=sr_1_7?ie=UTF8&qid=1486829376&sr=8-7&keywords=R+text+processing
• Pro Bash Programming: Scripting the Linux Shell (Expert's Voice in Linux) https://www.amazon.com/Pro-Bash-Programming-Scripting-Experts/dp/1430219971/ref=sr_1_fkmr0_2?ie=UTF8&qid=1486857179&sr=8-2-fkmr0&keywords=bash+text+processing
• Python Scripting for Computational Science: 3 (Texts in Computational Science and Engineering) https://www.amazon.com/Python-Scripting-Computational-Science-Engineering-ebook/dp/B001NLKSSO/ref=mt_kindle?_encoding=UTF8&me=
• Network Programmability and Automation: Skills for the Next-Generation Network Engineer https://www.amazon.com/Network-Programmability-Automation-Next-Generation-Engineer/dp/1491931256/ref=sr_1_17?ie=UTF8&qid=1487107643&sr=8-17&keywords=python+scripting
• Taming Text: How to Find, Organize, and Manipulate It https://www.safaribooksonline.com/library/view/taming-text-how/9781933988382/

bash

• Wicked Cool Shell Scripts, 2nd Edition https://www.safaribooksonline.com/library/view/wicked-cool-shell/9781492018322/

ruby

• Text Processing with Ruby https://www.safaribooksonline.com/library/view/text-processing-with/9781680501575/
• Practical Ruby for System Administration https://www.amazon.com/Practical-System-Administration-Experts-Source/dp/1590598210/ref=sr_1_304?ie=UTF8&qid=1487114901&sr=8-304&keywords=python+cloud
• Everyday Scripting with Ruby: For Teams, Testers, and You 1st Ed https://www.amazon.com/Everyday-Scripting-Ruby-Teams-Testers/dp/0977616614/ref=pd_sim_14_1?_encoding=UTF8&pd_rd_i=0977616614&pd_rd_r=09RA80YPVNPHFX6W9HRC&pd_rd_w=fpkVD&pd_rd_wg=xTaj2&psc=1&refRID=09RA80YPVNPHFX6W9HRC
• Wicked Cool Ruby Scripts: Useful Scripts that Solve Difficult Problems 1st Ed https://www.amazon.com/Wicked-Cool-Ruby-Scripts-Difficult/dp/1593271824/ref=pd_sim_14_4?_encoding=UTF8&pd_rd_i=1593271824&pd_rd_r=3NVCPNJ5PT5MGVZ7XB8X&pd_rd_w=pDh4w&pd_rd_wg=7Nm4C&psc=1&refRID=3NVCPNJ5PT5MGVZ7XB8X

python

• Python Quick Start for Linux System Administrators https://app.pluralsight.com/library/courses/python-linux-system-administrators/table-of-contents
• The Hitchhiker's Guide to Python: Best Practices for Development https://www.amazon.com/Hitchhikers-Guide-Python-Practices-Development/dp/1491933178/ref=pd_sim_14_12?_encoding=UTF8&pd_rd_i=1491933178&pd_rd_r=KEAW97J3NQEXG21CKWBY&pd_rd_w=QIwyC&pd_rd_wg=pzB4O&psc=1&refRID=KEAW97J3NQEXG21CKWBY
• Python Data Science Handbook: Essential Tools for Working with Data https://www.amazon.com/Python-Data-Science-Handbook-Essential/dp/1491912057/ref=pd_sim_14_3?_encoding=UTF8&pd_rd_i=1491912057&pd_rd_r=KEAW97J3NQEXG21CKWBY&pd_rd_w=QIwyC&pd_rd_wg=pzB4O&psc=1&refRID=KEAW97J3NQEXG21CKWBY
• Data Visualization with Python and JavaScript: Scrape, Clean, Explore & Transform Your Data https://www.amazon.com/Data-Visualization-Python-JavaScript-Transform/dp/1491920513/ref=pd_sim_14_4?_encoding=UTF8&pd_rd_i=1491920513&pd_rd_r=WAW4GKP39JZJPXKYZ54H&pd_rd_w=2BBbQ&pd_rd_wg=oYNJw&psc=1&refRID=WAW4GKP39JZJPXKYZ54H
• Mining the Social Web: Data Mining Facebook, Twitter, LinkedIn, Google+, GitHub, and More https://www.amazon.com/Mining-Social-Web-Facebook-LinkedIn/dp/1449367615/ref=sr_1_160?ie=UTF8&qid=1487114754&sr=8-160&keywords=python+cloud
• Web Scraping with Python: Collecting Data from the Modern Web https://www.amazon.com/Web-Scraping-Python-Collecting-Modern/dp/1491910291/ref=pd_bxgy_14_img_2?_encoding=UTF8&pd_rd_i=1491910291&pd_rd_r=4J8EDG8PV293H5RYSV8E&pd_rd_w=piDKd&pd_rd_wg=aqY1t&psc=1&refRID=4J8EDG8PV293H5RYSV8E
• Building Data Pipelines with Python https://www.safaribooksonline.com/library/view/building-data-pipelines/9781491970270/
• Fluent Python: Clear, Concise, and Effective Programming https://www.amazon.com/Fluent-Python-Concise-Effective-Programming/dp/1491946008/ref=pd_bxgy_14_img_2?_encoding=UTF8&pd_rd_i=1491946008&pd_rd_r=BDT30TJXXE3E6CNWNQ87&pd_rd_w=Sf9I5&pd_rd_wg=HjtNo&psc=1&refRID=BDT30TJXXE3E6CNWNQ87<- nicee!
• Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPythonhttps://www.amazon.com/Python-Data-Analysis-Wrangling-IPython/dp/1449319793/ref=sr_1_4?ie=UTF8&qid=1487114575&sr=8-4&keywords=python+cloud
• Python for Data Science For Dummies https://www.amazon.com/Python-Data-Science-Dummies-Computers-ebook/dp/B00TWK3RHW/ref=mt_kindle?_encoding=UTF8&me=
• Python: Visual QuickStart Guide, Third Edition https://www.safaribooksonline.com/library/view/python-visual-quickstart/9780133435160/ch11.html
• Python Essential Reference (4th Edition) https://www.amazon.com/Python-Essential-Reference-David-Beazley/dp/0672329786/ref=pd_bxgy_14_img_3?_encoding=UTF8&pd_rd_i=0672329786&pd_rd_r=QJ4H03MEJM726QZG56QZ&pd_rd_w=ZKBa5&pd_rd_wg=Btaz5&psc=1&refRID=QJ4H03MEJM726QZG56QZ
• Python in Practice: Create Better Programs Using Concurrency, Libraries, and Patterns (Developer's Library) https://www.amazon.com/Python-Practice-Concurrency-Libraries-Developers/dp/0321905636/ref=sr_1_106?ie=UTF8&qid=1487114691&sr=8-106&keywords=python+cloud
• Data Science from Scratch: First Principles with Python https://www.amazon.com/Data-Science-Scratch-Principles-Python/dp/149190142X/ref=pd_bxgy_14_img_3?_encoding=UTF8&pd_rd_i=149190142X&pd_rd_r=WAW4GKP39JZJPXKYZ54H&pd_rd_w=0j9ZK&pd_rd_wg=oYNJw&psc=1&refRID=WAW4GKP39JZJPXKYZ54H
• Data Wrangling with Python: Tips and Tools to Make Your Life Easier - REST vs Streaming API https://www.amazon.com/Data-Wrangling-Python-Tools-Easier/dp/1491948817/ref=pd_sim_14_6?_encoding=UTF8&pd_rd_i=1491948817&pd_rd_r=KEAW97J3NQEXG21CKWBY&pd_rd_w=QIwyC&pd_rd_wg=pzB4O&psc=1&refRID=KEAW97J3NQEXG21CKWBY

• Python for Unix and Linux System Administration 1st Ed https://www.amazon.com/Python-Unix-Linux-System-Administration/dp/0596515820/ref=sr_1_4?ie=UTF8&qid=1487708727&sr=8-4&keywords=python+system+administration
• Pro Python System Administration 1st Ed https://www.amazon.com/Python-System-Administration-Experts-Source/dp/1430226056/ref=sr_1_3?ie=UTF8&qid=1487708727&sr=8-3&keywords=python+system+administration
• Pro Python System Administration 2nd ed https://www.amazon.com/Python-System-Administration-Rytis-Sileika/dp/148420218X/ref=sr_1_1?ie=UTF8&qid=1487708727&sr=8-1&keywords=python+system+administration
• Foundations of Python Network Programming 3rd ed https://www.amazon.com/Foundations-Python-Network-Programming-Brandon/dp/1430258543/ref=pd_bxgy_14_img_2?_encoding=UTF8&pd_rd_i=1430258543&pd_rd_r=MGTKMKGFA3SGN0Z8XW2T&pd_rd_w=ppXt3&pd_rd_wg=gKBwL&psc=1&refRID=MGTKMKGFA3SGN0Z8XW2T

hadoop

• Data Analytics with Hadoop: An Introduction for Data Scientists https://www.amazon.com/Data-Analytics-Hadoop-Introduction-Scientists/dp/1491913703/ref=pd_sim_14_18?_encoding=UTF8&pd_rd_i=1491913703&pd_rd_r=KEAW97J3NQEXG21CKWBY&pd_rd_w=QIwyC&pd_rd_wg=pzB4O&psc=1&refRID=KEAW97J3NQEXG21CKWBY

spark

• Advanced Analytics with Spark: Patterns for Learning from Data at Scale https://www.amazon.com/Advanced-Analytics-Spark-Patterns-Learning/dp/1491912766/ref=pd_sim_14_27?_encoding=UTF8&pd_rd_i=1491912766&pd_rd_r=KEAW97J3NQEXG21CKWBY&pd_rd_w=QIwyC&pd_rd_wg=pzB4O&psc=1&refRID=KEAW97J3NQEXG21CKWBY

regex

• Regular Expression Recipes: A Problem-Solution Approach https://www.amazon.com/Regular-Expression-Recipes-Problem-Solution-Approach/dp/159059441X/ref=sr_1_189?ie=UTF8&qid=1487114792&sr=8-189&keywords=python+cloud

bad data

• Bad Data Handbook: Cleaning Up The Data So You Can Get Back To Work https://www.amazon.com/Bad-Data-Handbook-Cleaning-Back/dp/1449321887/ref=sr_1_272?ie=UTF8&qid=1487114866&sr=8-272&keywords=python+cloud

tpc-H

tpc-DS

tpc-H vs tpc-DS

Behold the TPC-DS, a new weapon in the global Big Data war
Seeking the 'game-proof' benchmark

There isn’t anything inherently evil about industry standard benchmarks, just as there isn’t anything inherently evil about guns. You know the saying: “Guns don’t kill people – people kill people.” (What about bullets? No, they’re not inherently evil either.)

But in the hands of motivated vendors, benchmarks are weapons to be wielded against competitors with great gusto. So why am I writing about benchmarks? It’s because the Transaction Processing Council (TPC) has released a new major benchmark, TPC-DS, which aims to provide a level playing field for vendors warring over their Big Data prowess.

El Reg's Timothy Prickett Morgan wrote about this new benchmark at length here. In his inimitable way, Tim captured the meat of the story, plus the marrow, gristle, hooves and hide of it too.

I talked with TPC reps last week about TPC-DS and benchmarks in general – learning quite a bit along the way about how TPC-DS is different (and better) than what we’ve seen before. In my years on the vendor side of the industry, there was nothing better than using a shiny new TPC-C or TPC-D score to smite the living hell out the server competition. But my efforts to learn enough to explain the results in customer presentations taught me bit about how vendors perform benchmarks. It’s like learning how sausage is made, but with more acronyms.

There were lots and lots of ways to put together stellar benchmark results on system configurations that a real customer would never buy, with software optimised in ways that won’t work in production environments, all offered at prices they’ll never see.

But benchmarks are a necessary evil, like the Irish and the Dutch. There are differences between systems and the ways vendors implement similar or even identical technology. Two Xeon processors might perform exactly the same when they roll out of Intel’s fab. But after they’ve been incorporated into various ‘industry standard’ systems, get their software loads and start running apps, there will be differences in performance and certainly price/performance.

In a perfect world, customers would be able to do ‘bake-offs’ or ‘try and buy’ exercises before issuing a PO. But these are time-consuming and expensive for both the vendor and customer. It’s worth the effort for a very large or mission-critical installation, but not so much when the deal size and importance is lesser. This is where standard benchmarks come in: they give buyers a few more data points to use in their decision making process.

Benchmark organisations like TPC and SPEC (Standard Performance Evaluation Council) arose from industry need for standardized and valid comparisons between systems and solutions. These consortiums are primarily vendor-funded, and vendors certainly play a role in helping shape the benchmarks.

If a benchmark's results are printed in a forest and no one reads them, does it exist?
Building standard benchmarks isn’t easy. Tests need to stress the right parts of the system/solution – mimicking real-world processing as much as possible. They need to be complex enough to stress even the largest system, but at the same time scalable and reasonably easy and inexpensive to run. Safeguards have to be built in so that vendors can’t rig the results or game the tests. And, finally, the rules need to be documented and enforced.

But for a benchmark to be truly successful, it has to be used by vendors and customers alike.

What jumped out during my briefing with TPC was the size and complexity of TPC-DS. Lots of tables, lots of data, and lots of operations. It’s much more complete and complex than TPC-H, with 99 queries compared to only 22 for the venerable H.

One of the newest wrinkles with DS is that the ‘ad hoc’ queries are truly ad hoc. In TPC-D and, to a lesser extent, TPC-H, ad hoc queries could be anticipated and planned for by canny benchmark engineers. With DS, the ad hoc queries are randomized, and there are too many potential permutations to allow for pre-positioning data accurately.

The TPC-DS‘number’ is a bit complicated to understand – it’s a combination of the single user time to run through 99 queries, multiple users running through 99 queries, and database load times. A valid TPC-DS run has to have the single user run results plus a multi-user run with at least four streams.

There isn’t a maximum number of streams (which is the same as users) that need to be tested. There also is no minimum response time goal that needs to be satisfied in the multi-user runs, like in other benchmarks. The metric is heavily influenced by the multi-user results, rewarding scalability and optimised use of shared resources.

Given that there aren’t maximum response time rules, vendors will end up doing lots of TPC-DS multi-user runs in order to find the sweet spot where they’re supporting a large number of users and achieving their best overall score. The executive summary of the mandatory benchmark disclosure document shows the numbers that went into the overall metric, including number of users supported, response times, and the like.

To me, TPC-DS is a big leap forward. I’m no benchmarking expert, and nothing is totally foolproof, but this looks to be a highly ‘game-proof’ benchmark. It also does a much better job of capturing the complexity of modern business processing to a much greater extent than its predecessors.

From a customer perspective, it’s definitely worth consulting TPC-DS before making that next Big Data solution buy. Now we wait and see if the vendors are going to run it through some gear – and thus weaponise it. ®

pm2

socket programming

programming

logic and algorithms

Creating AFCS file systems on Exadata storage requires the following:

Oracle Linux
Grid Infrastructure 12.1.0.2
Database files stored in ACFS on Exadata storage are subject to the following guidelines and restrictions:

Supported database versions are 10.2.0.4, 10.2.0.5, 11.2.0.4, and 12.1.
Hybrid Columnar Compression (HCC) support (for 11.2 and 12.1) requires fix for bug 19136936.
Exadata-offload features such as Smart Scan, Storage Indexes, IORM, Network RM, etc. are not supported.
Exadata Smart Flash Cache will cache read operations. Caching of write operations is expected in a later release.
No specialized cache hints are passed from the Database to the Exadata Storage layer, which means the Smart Flash Cache heuristics are based on I/O size, similar to any other block storage caching technology.
Exadata Smart Flash Logging is not supported.
Hardware Assisted Resilient Data (HARD) checks are not performed.