Tokens

Tokens

Python breaks each logical line into a sequence of elementary lexical components known as Tokens. The normal token types are

1) Identifiers,

2) Keywords,

3) Operators,

4) Delimiters and

5) Literals.

Whitespace separation is necessary between tokens, identifiers or keywords.

1) Identifiers

An Identifier is a name used to identify a variable, function, class, module or object.

·        An identifier must start with an alphabet (A..Z or a..z) or underscore ( _ ).

·        Identifiers may contain digits (0 .. 9)

·        Python identifiers are case sensitive i.e. uppercase and lowercase letters are distinct.

·        Identifiers must not be a python keyword.

·        Python does not allow punctuation character such as %,$, @ etc., within identifiers.

Example of valid identifiers

Sum, total_marks, regno, num1

Example of invalid identifiers

12Name, name$, total-mark, continue

2. Keywords

Keywords are special words used by Python interpreter to recognize the structure of program. As these words have specific meaning for interpreter, they cannot be used for any other purpose.    

3. Operators

In computer programming languages operators are special symbols which represent computations, conditional matching etc. The value of an operator used is called operands. Operators are categorized as Arithmetic, Relational, Logical, Assignment etc. Value and variables when used with operator are known as operands.

(i) Arithmetic operators

An arithmetic operator is a mathematical operator that takes two operands and performs a calculation on them. They are used for simple arithmetic. Most computer languages contain a set of such operators that can be used within equations to perform different types of sequential calculations.

Python supports the following Arithmetic operators.

Operator - Operation             Examples   Result        

Assume a=100 and b=10. Evaluate the following expressions

+ (Addition)                   >>> a + b    =110

- (Subtraction)               >>>a – b     =90

* (Multiplication)           >>> a*b      =1000

/ (Divisioin)          >>> a / b     =10.0

% (Modulus)                  >>> a % 30 =10

** (Exponent)                >>> a ** 2  =10000

// (Floor Division)          >>> a//30 (Integer Division)    =3

Program 5.1 To test Arithmetic Operators:

#Demo Program to test Arithmetic Operators

a=100

b=10

print ("The Sum= ",a+b)

print ("The Difference = ",a-b)

print ("The Product= ",a*b)

print ("The Quotient  = ",a/b)

print ("The Remainder = ",a%30)

print ("The Exponent  = ",a**2)

print ("The Floor Division =",a//30)

#Program End

Output:

The Sum= 110

The Difference= 90

The Product= 1000

The Quotient= 10.0

The Remainder= 10

The Exponent= 10000

The Floor Division= 3

(ii) Relational or Comparative operators

A Relational operator is also called as Comparative operator which checks the relationship between two operands. If the relation is true, it returns True; otherwise it returns False.

Python supports following relational operators

Coding 5.2 To test Relational Operators:

#Demo Program to test Relational Operators

a=int (input("Enter a Value for A:"))

b=int (input("Enter a Value for B:"))

print ("A = ",a," and B = ",b)

print ("The a==b = ",a==b)

print ("The a > b = ",a>b)

print ("The a < b = ",a<b)

print ("The a >= b = ",a>=b)

print ("The a <= b = ",a<=0)

print ("The a != b = ",a!=b)

#Program End

Output:

Enter a Value for A:35

Enter a Value for B:56

A = 35 and B= 56

The a==b= False

The a > b= False

The a < b= True

The a >= b= False

The a <= b= False

The a != b= True

(iii) Logical operators

In python, Logical operators are used to perform logical operations on the given relational expressions. There are three logical operators they are and, or and not.

Program 5.3 To test Logical Operators:

Example – Code

#Demo Program to test Logical Operators

a=int (input("Enter a Value for A:"))

b=int (input("Enter a Value for B:"))

print ("A = ",a, " and b = ",b)

print ("The a > b or a == b = ",a>b or a==b)

print ("The a > b and a == b = ",a>b and a==b)

print ("The not a > b = ",not a>b)

#Program End

Example - Result

Enter a Value for A:50

Enter a Value for B:40

A = 50 and b = 40

The a > b or a == b = True

The a > b and a == b = False

The not a > b = False

(iv) Assignment operators

In Python, = is a simple assignment operator to assign values to variable. Let a = 5 and b = 10 assigns the value 5 to a and 10 to b these two assignment statement can also be given as a,b=5,10 that assigns the value 5 and 10 on the right to the variables a and b respectively. There are various compound operators in Python like +=, -=, *=, /=, %=, **= and //= are also available.

(v) Conditional operator

Ternary operator is also known as conditional operator that evaluate something based on a condition being true or false. It simply allows testing a condition in a single line replacing the multiline if-else making the code compact.

The Syntax conditional operator is,

Variable Name = [on_true] if [Test expression] else [on_false]

Example :

min= 50 if 49<50 else 70 # min = 50

min= 50 if 49>50 else 70 # min = 70

Program 5.5 To test Conditional (Ternary) Operator:

#Program to demonstrate conditional operator

a, b = 30, 20

#Copy value of a in min if a < b else copy b

min = a if a < b else b

print ("The Minimum of A and B is ",min)

# End of the Program

Output:

The Minimum of A and B is 20

4. Delimiters

Python uses the symbols and symbol combinations as delimiters in expressions, lists, dictionaries and strings. Following are the delimiters.

5. Literals

Literal is a raw data given in a variable or constant. In Python, there are various types of literals.

1. Numeric

2. String

3. Boolean

(i) Numeric Literals

Numeric Literals consists of digits and are immutable (unchangeable). Numeric literals can belong to 3 different numerical types Integer, Float and Complex.

Program 5.6 : To demonstrate Numeric literals

# Program to demonstrate Numeric Literals

a = 0b1010 #Binary Literals

b = 100       #Decimal Literal

c = 0o310   #Octal Literal

d = 0x12c   #Hexadecimal Literal

print ("Integer Literals :",a,b,c,d)

#Float Literal

float_1 = 10.5

float_2 = 1.5e2

print ("Float Literals :",float_1,float_2)

#Complex Literal

x = 1 + 3.14 j

print ("Complex Literals :", x)

Print ("x = ", x , "Imaginary part of x = ", x.imag, "Real part of x = ", x.real)

#End of the Program

Output:

Integer Literals : 10 100 200 300

Float Literals : 10.5 150.0

Complex Literals :

x = (1.3.14) Imaginary part of x = 3.14 Real part of 9 x = 1.0

(ii) String Literals

In Python a string literal is a sequence of characters surrounded by quotes. Python supports single, double and triple quotes for a string. A character literal is a single character surrounded by single or double quotes. The value with triple-quote "' '" is used to give multi-line string literal.

Program 5.7 To test String Literals

#Demo Program to test String Literals

strings = "This is Python"

char = "C"

multiline_str = "'This is a multiline string with more than one line code."'

print (strings)

print (char)

print (multiline_str)

#End of the Program

Output:

This is Python

C

This is a multiline string with more than one line code.

(iii) Boolean Literals

A Boolean literal can have any of the two values: True or False.

Program 5.8 To test Boolean Literals:

#Demo Program to test String Literals

boolean_1 = True

boolean_2 = False

print ("Demo Program for Boolean Literals")

print ("Boolean Value1 :",boolean_1)

print ("Boolean Value2 :",boolean_2)

#End of the Program

Output:

Demo Program for Boolean Literals

Boolean Value1 : True

Boolean Value2 : False

(iv) Escape Sequences

In Python strings, the backslash "\" is a special character, also called the "escape" character. It is used in representing certain whitespace characters: "\t" is a tab, "\n" is a newline, and "\r" is a carriage return. For example to print the message "It's raining", the Python command is

>>>print ("It\'Sn rainning")

It's ranning

Python supports the following escape sequence characters.