COMP1531 Terminalogies

Note: Based on python

WEEK4

Data Interchange Formats

When it comes to transferring data, we also need common interface(s) that people all send or store data in universal ways to be shared between applications or shared over networks.

JSON

A format made up of braces for dictionaries, square brackets for lists, where all non-numeric items must be wrapped in quotations. Very similar to python data structures.

{
	"locations": [
    	{
            "suburb" : "Kensington",
            "postcode" : 2033
        },
        {
            "suburb" : "Mascot",
            "postcode" : 2020
        },
        {
            "suburb" : "Sydney CBD",
            "postcode" : 2000
        }
    ]
}

YAML

YAML Ain’t Markup Language (YAML) is a popular modern interchange format due it’s ease of editing and concise nature. It’s easy to convert between JSON and YAML online.

locations:
 - suburb: Kensington
  postcode: 2033
 - suburb: Mascot
  postcode: 2020
 - suburb: Sydney CBD
  postcode: 2000

XML

eXtensible Markup Language (XML) is more of a legacy interchange format being used less and less

<?xml version="1.0" encoding="UTF-8"?>
<root>
   <locations>
      <element>
         <postcode>2033</postcode>
         <suburb>Kensington</suburb>
      </element>
      <element>
         <postcode>2020</postcode>
         <suburb>Mascot</suburb>
      </element>
      <element>
         <postcode>2000</postcode>
         <suburb>Sydney CBD</suburb>
      </element>
   </locations>
</root>

Issues with XML include:

• More verbose (harder to read at a glance)
• More demanding to process/interpret
• More bytes required to store (due to open/closing tags)
  While you will find very few modern applications choose to use XML as an interchange format, many legacy systems will still use XML as a means of storing data

Flask

Computer Networks

• Network: A group of interconnected computers that can communicate
• Internet: A global infrastructure for networking computers around the entire world together
• World Wide Web: A system of documents and resources linked together, accessible via URLs

Network Protocols

Example

HTTP - Hypertext Transfer Protocol

I.E. Protocol for sending and receiving HTML documents (nowadays much more)
HTTP Request

GET /hello HTTP/1.1
Host: 127.0.0.1:5000
Connection: keep-alive
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
Sec-Fetch-Site: none
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8

HTTP Response

HTTP/1.0 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 12
Server: Werkzeug/0.16.0 Python/3.5.3
Date: Wed, 09 Oct 2019 13:21:51 GMT
 
Hello world!

Flask

Normal message

from flask import Flask
APP = Flask(__name__)
 
@APP.route("/")
def hello():
    return "Hello World!"
 
if __name__ == "__main__":
    APP.run()

Serve an Image

	
from flask import Flask, send_file
APP = Flask(__name__)
 
@APP.route("/img")
def img():
    return send_file('./cat.jpg', mimetype='image/jpg')
 
if __name__ == "__main__":
    APP.run()

API

An API (Application Programming Interface) refers to an interface exposed by a particular piece of software.
The most common usage of “API” is for Web APIs, which refer to a “contract” that a particular service provides. The interface of the service acts as a black box and indicates that for particular endpoints, and given particular input, the client can expect to receive particular output.
Web API

Restful API & “CRUD”

A RESTful API is an application program interface (API) that uses HTTP requests to GET, PUT, POST and DELETE data. These 4 methods describe the “nature” of different API requests.

from flask import Flask, request
from json import dumps
 
APP = Flask(__name__)
 
@APP.route("/one", methods=['GET'])
def one():
	# json.dumps(data, file) could dumps the data to a new opened file
    return dumps({
    	'1': request.args.get('data1'),
    	'2': request.args.get('data1'),
    })
 
@APP.route("/two", methods=['POST'])
def two():
    data = request.get_json()
    return dumps({
    	'1': data['data1'],
    	'2': data['data2'],
    })
 
if __name__ == '__main__':
    APP.run()

API Client

echo.py

from flask import Flask, request
from json import dumps
 
APP = Flask(__name__)
 
@APP.route("/echo", methods=['GET'])
def echo():
    return dumps({'data': request.args.get('data')})
 
if __name__ == '__main__':
    APP.run()

echo_main.py

import json
import requests
 
def get_payload():
    response = requests.get('http://127.0.0.1:5000/echo', params={'data': 'hi'})
    payload = response.json()
    print(payload)
 
if __name__ == '__main__':
    get_payload()

WEEK5

Authentication vs. Authorisation

Authentication: Process of verifying the identity of a user
Authorisation: Process of verifying an identity’s access privileges

Strength Authentication - hashlib

import hashlib
print("mypassword")
print("mypassword".encode())
print(hashlib.sha256("mypassword".encode()))
print(hashlib.sha256("mypassword".encode()).hexdigest())
###################################################################
# output
mypassword
b'mypassword'
<sha256 HASH object @ 0x000001D5D7FCFAB0>
89e01536ac207279409d4de1e5253e01f4a1769e696db0d6062ca9b8f56767c8

The point is as the key is hashed, it cannot be reversed!!!

Strength Authorisation - JWT

Play with JWT

import jwt
SECRET = 'sempai'
encoded_jwt = jwt.encode({'some': 'payload'}, SECRET, algorithm='HS256').decode('utf-8')
print(encoded_jwt)
print(jwt.decode(encoded_jwt.encode('utf-8'), SECRET, algorithms=['HS256']))
#################################################################################################
# output
eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzb21lIjoicGF5bG9hZCJ9.RB9IITs-Nwr98yYtOwpiZuRD1nQ8ST_SJZF4Ni0-ndg
{'some': 'payload'}

The point is as the key is hashed, it can be decoded!!!

SDLC Design - Software Engineering Principles

Design Smell

• Rigidity: Tendency to be too difficult to change
• Fragility: Tendency for software to break when single change is made
• Immobility: Previous work is hard to reuse or move
• Viscosity: Changes feel very slow to implement
• Opacity: Difficult to understand
• Needless complexity: Things done more complex than they should be
• Needless repetition: Lack of unified structures
• Coupling: Interdependence between components

Design Principles

Purpose is to make items:

• Extensible
• Reusable
• Maintainable
• Understandable
• Testable
  Often, this is achieved through abstraction. Abstraction is the process of removing characteristics of something to reduce it some a more high level concept.

DRY - Don’t repeat yourself

“Don’t repeat yourself” (DRY) is about reducing repetition in code. The same code/configuration should ideally not be written in multiple places.

DLC Development - Persistence

Data

Data: Facts that can be recorded and have implicit meaning
From data (raw) was can find insights (information) that allow us to make decisions.

Data Layers

Data Layer: The part of the tech stack that provides persistence

Database

There are 3 main ways to store data:

• In-memory (non-persistent)
• In-file
• In-database (SQL)

Storing Data: Persistence

Persistence: When program state outlives the process that created it. This is achieved by storing the state as data in computer data storage
Most modern backend/server applications are just source code + data

Example: In-file

import pickle
with open('datastore.p', 'wb') as FILE:
	pickle.dump(data, FILE)
###########################################
# output
This will generate a new file called 'datastore.p' store the bunch of binary data

import pickle
DATA = pickle.load(open('datastore.p', 'rb'))
###########################################
# output
This will read the binary data from a file called 'datastore.p' and pass the data to DATA parm

pickle
Pickle: The pickle module implements binary protocols for serializing and de-serializing a python object structure.

WEEK7

Develop a plan for the product - Requirement Analysis

Requirements

User Stories

User Stories are a method of requirements engineering used to inform the development process and what features to build with the user at the centre.
i.e. As a < type of user >, I want < some goal > so that < some reason >
e.g. As a student, I want to purchase a parking pass so that I can drive to school

THINK LIKE AN ENGENDER !

INVEST (the way to meet users stories)

I = Independent: user story could be developed independently and delivered separately
N = Negotiable: avoid too much detail.
V = Valuable: must hold some value to the client
E = Estimable: the requirements should be meet in the limited time
S = Small: user story should be small
T = Testable

User Acceptance Criteria

Break down a user story into criteria that must be met for the user, or customer, to accept
Written in natural language
e.g. As a user, I want to use a search field to type a city, name, or street, so that I can find matching hotel options.

• The search field is placed on the top bar
• Search starts once the user clicks “Search”
• The field contains a placeholder with a grey-colored text: “Where are
  you going?”
• The placeholder disappears once the user starts typing
• Search is performed if a user types in a city, hotel name, street, or
  all combined
• The user can’t type more than 200 symbols

From Criteria to Testing

Acceptance Tests are tests that are performed to ensure acceptance criteria have been met
Not all acceptance criteria can easily be mapped to automated acceptance tests
Acceptance tests are black-box tests
e.g. As a user, I can log in through a social media account, because I always forget my passwords

• Can log in through Facebook
• Can log in through LinkedIn
• Can log in through Twitter

Scenario Oriented AC

The Acceptance criteria from before are often referred to a rule-based AC
Sometimes it is preferable to have AC that describe a scenario
This can be done in the Given/When/Then format:

• Given some precondition
• When I do some action
• Then I expect some result
  e.g. As a user, I want to be able to recover the password to my account, so that I will be able to access my account in case I forgot the password.
  Scenario: Forgot password
  Given: The user has navigated to the login page
  When: The user selected forgot password option
  And: Entered a valid email to receive a link for password recovery
  Then: The system sent the link to the entered email
  Given: The user received the link via the email
  When: The user navigated through the link received in the email
  Then: The system enables the user to set a new password

Summary for three types above

• Rule-based acceptance criteria are simpler and generally work for all
  sorts of stories
• Scenario-based AC work for stories that imply specific user actions,
  but don’t work for higher-level system properties (e.g. design)
• Scenario-based AC are more likely to be implementable as tests

SDLC Design - System Modelling

Conceptual Modelling

A model of a concept
Purpose:

• To predict future states of affairs.
• Understand the current state of affairs.
• Determine the past state of affairs.
• To convey the fundamental principles and basic functionality of
  systems (communication)

Communicating models

Four fundamental objectives of communicating with a conceptual model:

• Enhance an individual’s understanding of the representative system
• Facilitate efficient conveyance of system details between
  stakeholders
• Provide a point of reference for system designers to extract system
  specifications
• Document the system for future reference and provide a means for
  collaboration

System Modelling

Structural

Emphasise the static structure of the system
e.g.

• UML class diagrams
• ER diagrams
• … many others

Behavioural

Emphasise the dynamic behaviour
e.g.

• State diagrams
• Use case diagram
• … some others

State diagrams

A diagrammatic representation of a state.
Some variation in notation.
Typically: states are circles, transitions are labelled arrows connecting them3
i.e.

e.g.

WEEK8

Decorators

def decorator(functinon):
	def manipulation(*args, **kargs):
		return function(*args, **kargs)
	return manipulation

Single Responsibility Principle

One function for one function, it shouldn’t do too many things

Deployment

Continues Integration / Continues Delivery

Continues Integration: Practice of automating the integration of code changes from multiple contributors into a single software project.

Key principles and processes

• Write tests:
  a. Ideally tests for each story
  b. Broad tests: unit, integration, acceptance, UI tests
• Use code coverage checkers
• Merge and integrate code as often as possible
• Ensure the build always works (i.e. is “green”)

How it works

Typically tests will be run by a “runner”, which is an application that works with your version control software (git) to execute the tests. This is because tests can require quite resource intensive activities
Gitlab: No runners built in
Bitbucket: Runners built in

CI/CD relationship

3 tiers of deployment

• dev: release often, avaliable to developers to see their changes in
  deployment
• test: as close to release as possible, ideally identical to prod
• prod: released to customers, ideallly as quick as possible

Flighting

Flighting is a term used predominately in larger software projects to describe moving builds out to particular slices of users, beyond the simplicity of “dev”, “test”, “prod”

Flighting Example - Windows

Continues Deployment

Continuous Deployment is an extension of Continuous Delivery whereby changes attempt to flight toward production automatically, and the only thing stopping them is a failed test

Comparing Continues Delivery and Continuous Deployment

DevOps (for fun)

modern DevOps is less a role, and more a series of roles or aspect of a role.

Monitoring and Maintenance(for fun)

After deployment, the use of analytics and monitoring tools to ensure that as the platform is used and remains in a healthy, Health is defined by developers, but often consists of: Monitoring 4XX and 5XX errors or Ensuring disk, memory, cpu, and network is not overloadedc

Two purposes of monitoring

Preserving user experience

Monitoring errors, warnings, and other issues that affect performance or uptime. (To keep users’ experience good)

Enhancing user experience

Using analytical tools to monitor users or understanding their interactions. Often leads to customer interviews and user stories (Monitoring the performance of the program and make the further improvement, knowing what’s the next requirement might be)

Week10

Iterators

In python, iterators and objects containing a countable number of elements

animals = ['Dogs', 'Cats', 'Sheep', 'Fish']
# iterable:
print(animals)
print(type(animals))
print(iter(animals))
print(type(iter(animals)))
#####################
# output
['Dogs', 'Cats', 'Sheep', 'Fish']
<class 'list'>
<list_iterator object at 0x7f0f19c55400>
<class 'list_iterator'>

iterator class

class Squares:
	def __init__(self):
		self.i = 0
	def __iter__(self):
		return self
	def __next__(self):
		self.i += 1
		return self.i*self.i

class Squares:
	def __init__(self, Maxbase):
		self.i = 0
		self.max = Maxbase
	def __iter__(self):
		return self
	def __next__(self, Maxbase):
		if self.i >= self.max:
			raise StopIteration # built-in functions
		self.i += 1
		return self.i*self.i

Generators

Intuitively, you can think of a generator as a suspendable computation
Calling next() on a generator executes it until it reaches a yield, at which point it is suspended (frozen) until the subsequent call to next()

def simple_generator():
	print("Hello")
	yield 1   # function suspend here
	print("Nice to meet you")
	yield 2   # suspend here again
	print("I am a generator")
####################
# output
Hello
1
Nice to meet you
2
I am a generator

rang() is actually a generator

Difference between generator and iterator

个人觉得这个文章里理解更透彻，贴上来: https://blog.csdn.net/shunelly/article/details/39352637

Libraries

Try not to reinvest wheels in python, try to import libraries. But the downside is the dependency will be higher as more you are developing.

Case study: leftpad

A Javascript library that had many users, mostly indirect
Owing to a disagreement, the author removed the library from NPM
This caused thousands of Javascript-based applications and libraries to break
(They just use the one simple function but import the whole library)