Friday, April 13, 2018

Thursday, April 05, 2018

Thoughts and an example on Expression-Oriented Programming

Here is the code, to read password from .pgpass to connect o a PostgreSQL database.

object DbUtil {
  def dbPassword(hostname:String, port:String, database:String, username:String ):String = {
    // Usage: val thatPassWord = dbPassword(hostname,port,database,username)
    // .pgpass file format, hostname:port:database:username:password
    val passwdFile = new java.io.File(scala.sys.env("HOME"), ".pgpass")
    var passwd = ""
    val fileSrc = scala.io.Source.fromFile(passwdFile)
    fileSrc.getLines.foreach{line =>
      val connCfg = line.split(":")
      if (hostname == connCfg(0)
        && port == connCfg(1)
        && database == connCfg(2)
        && username == connCfg(3)
      ) { 
        passwd = connCfg(4)
      }
    }
    fileSrc.close
    passwd
  }

  def passwordFromConn(connStr:String) = {
    // Usage: passwordFromConn("hostname:port:database:username")
    val connCfg = connStr.split(":")
    dbPassword(connCfg(0),connCfg(1),connCfg(2),connCfg(3))
  }
}

Thoughts.

* foreach v.s.  map().filter()(0) . e.g.: List(1,2,3).filter(_ > 0)(0)
* var passwd, v.s.  val passwd inside of foreach.
* is it good to do chain of method style?

* how to break when find one, in a for loop ?

import util.control.Breaks._
breakable {
for (i <- 1 to 10) {
println(i)
if (i > 4) break // break out of the for loop }
}

* other improvement ?

Wednesday, March 14, 2018

From Python chapter 2 code solution

Here is my answer for book:

从Python开始学编程


Chapter 2, Appendix A, Exam.

https://book.douban.com/subject/26919485/
..

from itertools import repeat

def gen():
int_rate = [0.01, 0.02, 0.03, 0.035, 0.05]
for r in int_rate:
yield r
for r in repeat(0.05):
yield r

def num_year_pay() :
house_price = 500000
year_pay = 30000
year_n = 1
rem = house_price

print(year_pay)

for r in gen():
if rem <=0:
break
print(r)
rem = rem * (1 + r) - year_pay
year_n = year_n + 1
print(rem)

return year_n

if __name__ == "__main__":
print("Number of years to pay loan: ", num_year_pay())

Number of years to pay loan: 31

Thursday, January 11, 2018

work with a custom schema with Spark loading CSV file

Here's how you can work with a custom schema, a complete demo:
Unix shell code,
$>
echo "
Slingo, iOS 
Slingo, Android
" > game.csv
Scala code:
import org.apache.spark.sql.types._

val customSchema = StructType(Array(
  StructField("game_id", StringType, true),
  StructField("os_id", StringType, true)
))

val csv_df = spark.read.format("csv").schema(customSchema).load("game.csv")
csv_df.show 

csv_df.orderBy(asc("game_id"), desc("os_id")).show
csv_df.createOrReplaceTempView("game_view")
val sort_df = sql("select * from game_view order by game_id, os_id desc")
sort_df.show 

Reference,