在这个lab中，需要为SimpleDB编写一组operators实现表数据的modifications（e.g., insert and delete records）、selections、joins和aggregates。
另外，我们在Lab1中忽视了一点：BufferPool里的numPages参数确定了读取的固定页数，lab中如果页的数量超过numPages，先不实现eviction policy，先扔出一个DbException错误。在lab2中，我们需要实现淘汰机制(eviction policy)。同时，这个lab中不需要实现transactions或locking。
1. Getting started
1.3. Implementation hints
下面是一个推进SimpleDB实现的简要指导：
- 实现Filter和Join操作符，并通过相应测试。这些操作符的Javadoc comments会有实现的细节，已经给出了Project和OrderBy的实现细节，这有助于你理解这些操作符是怎么工作的。
- 实现IntegerAggregator和StringAggregator。这里要写aggregate（分组聚合）的计算逻辑，在一系列输入的行中跨越多个groups寻找特定的列。使用整数除法计算平均值，因为SimpleDB只支持整数。StringAggegator只需要支持COUNT aggregate，因为其他操作符对于strings没有意义。
- 实现Aggregate操作符。像其他操作符一样，Aggregate实现了OpIterator接口，所以能被放入SimpleDB 查询策略(query plans)。要注意Aggregate操作符的输出是每调用一次next()输出整个group的aggregate value。
- 实现BufferPool和tuple insertion, deletion 和 page eviction有关的方法，在这里先不用关心transactions。
- 实现Insert和Delete操作符。像其他操作符一样，Insert和Delete实现了OpIterator，接收tuples数据流插入或删除，并输出一行整数指明插入或删除的tuples数量。这些操作符需要调用BufferPool中合适的方法，能够实际修改硬盘上的物理页内容。
注意SimpleDB没有实现任何一致性(consistency)和完整性(integrity)检测，所以是有可能插入重复记录，所以也没办法设置主键或外键约束。
这个lab的目标是通过systemtest中的测试。
2. SimpleDB Architecture and Implementation Guide
2.1. Filter and Join
SimpleDB的OpIterator实现了关系代数的操作符，现在为了更强的查询功能需要实现两个操作符。
- Filter:这个操作符只返回满足Predicate的tuples，也就是过滤掉不满足条件的。
- Join: 这个操作符根据JoinPredicate联结tuples，我们只需要简单的nested loops join。
Exercise 1.
Implement the skeleton methods in:
- src/simpledb/Predicate.java
- src/simpledb/JoinPredicate.java
- src/simpledb/Filter.java
- src/simpledb/Join.java
At this point, your code should pass the unit tests in PredicateTest, JoinPredicateTest, FilterTest, and JoinTest. Furthermore, you should be able to pass the system tests FilterTest and JoinTest.
src/simpledb/Predicate.java代码实现如下：
package simpledb;
import java.io.Serializable;
/**
* Predicate compares tuples to a specified Field value.
*/
public class Predicate implements Serializable {
private static final long serialVersionUID = 1L;
/** Constants used for return codes in Field.compare */
public enum Op implements Serializable {
EQUALS, GREATER_THAN, LESS_THAN, LESS_THAN_OR_EQ, GREATER_THAN_OR_EQ, LIKE, NOT_EQUALS;
/**
* Interface to access operations by integer value for command-line
* convenience.
*
* @param i
* a valid integer Op index
*/
public static Op getOp(int i) {
return values()[i];
}
public String toString() {
if (this == EQUALS)
return "=";
if (this == GREATER_THAN)
return ">";
if (this == LESS_THAN)
return "<";
if (this == LESS_THAN_OR_EQ)
return "<=";
if (this == GREATER_THAN_OR_EQ)
return ">=";
if (this == LIKE)
return "LIKE";
if (this == NOT_EQUALS)
return "<>";
throw new IllegalStateException("impossible to reach here");
}
}
private final int field;
private final Op op;
private final Field operand;
/**
* Constructor.
*
* @param field
* field number of passed in tuples to compare against.
* @param op
* operation to use for comparison
* @param operand
* field value to compare passed in tuples to
*/
public Predicate(int field, Op op, Field operand) {
// some code goes here
this.field = field;
this.op = op;
this.operand = operand;
}
/**
* @return the field number
*/
public int getField()
{
// some code goes here
return field;
}
/**
* @return the operator
*/
public Op getOp()
{
// some code goes here
return op;
}
/**
* @return the operand
*/
public Field getOperand()
{
// some code goes here
return operand;
}
/**
* Compares the field number of t specified in the constructor to the
* operand field specified in the constructor using the operator specific in
* the constructor. The comparison can be made through Field's compare
* method.
*
* @param t
* The tuple to compare against
* @return true if the comparison is true, false otherwise.
*/
public boolean filter(Tuple t) {
// some code goes here
return t.getField(field).compare(op,operand);
}
/**
* Returns something useful, like "f = field_id op = op_string operand =
* operand_string"
*/
public String toString() {
// some code goes here
String s = String.format("f = %d op = %s operand = %s", field,op.toString(),operand.toString());
return s;
}
}
src/simpledb/JoinPredicate.java代码实现如下：
package simpledb;
import java.io.Serializable;
/**
* JoinPredicate compares fields of two tuples using a predicate. JoinPredicate
* is most likely used by the Join operator.
*/
public class JoinPredicate implements Serializable {
private static final long serialVersionUID = 1L;
private final int field1;
private final int field2;
private final Predicate.Op op;
/**
* Constructor -- create a new predicate over two fields of two tuples.
*
* @param field1
* The field index into the first tuple in the predicate
* @param field2
* The field index into the second tuple in the predicate
* @param op
* The operation to apply (as defined in Predicate.Op); either
* Predicate.Op.GREATER_THAN, Predicate.Op.LESS_THAN,
* Predicate.Op.EQUAL, Predicate.Op.GREATER_THAN_OR_EQ, or
* Predicate.Op.LESS_THAN_OR_EQ
* @see Predicate
*/
public JoinPredicate(int field1, Predicate.Op op, int field2) {
// some code goes here
this.field1 = field1;
this.field2 = field2;
this.op = op;
}
/**
* Apply the predicate to the two specified tuples. The comparison can be
* made through Field's compare method.
*
* @return true if the tuples satisfy the predicate.
*/
public boolean filter(Tuple t1, Tuple t2) {
// some code goes here
return t1.getField(field1).compare(op,t2.getField(field2));
}
public int getField1()
{
// some code goes here
return field1;
}
public int getField2()
{
// some code goes here
return field2;
}
public Predicate.Op getOperator()
{
// some code goes here
return op;
}
}
src/simpledb/Filter.java代码实现如下：
package simpledb;
import java.util.*;
/**
* Filter is an operator that implements a relational select.
*/
public class Filter extends Operator {
private static final long serialVersionUID = 1L;
private final Predicate p;
private OpIterator child;
/**
* Constructor accepts a predicate to apply and a child operator to read
* tuples to filter from.
*
* @param p
* The predicate to filter tuples with
* @param child
* The child operator
*/
public Filter(Predicate p, OpIterator child) {
// some code goes here
this.p = p;
this.child = child;
}
public Predicate getPredicate() {
// some code goes here
return p;
}
public TupleDesc getTupleDesc() {
// some code goes here
return child.getTupleDesc();
}
public void open() throws DbException, NoSuchElementException,
TransactionAbortedException {
// some code goes here
child.open();
super.open();
}
public void close() {
// some code goes here
super.close();
child.close();
}
public void rewind() throws DbException, TransactionAbortedException {
// some code goes here
child.rewind();
}
/**
* AbstractDbIterator.readNext implementation. Iterates over tuples from the
* child operator, applying the predicate to them and returning those that
* pass the predicate (i.e. for which the Predicate.filter() returns true.)
*
* @return The next tuple that passes the filter, or null if there are no
* more tuples
* @see Predicate#filter
*/
protected Tuple fetchNext() throws NoSuchElementException,
TransactionAbortedException, DbException {
// some code goes here
while(child.hasNext()){
Tuple t = child.next();
if(p.filter(t)){
return t;
}
}
return null;
}
@Override
public OpIterator[] getChildren() {
// some code goes here
return new OpIterator[] {this.child};
}
@Override
public void setChildren(OpIterator[] children) {
// some code goes here
child = children[0];
}
}
src/simpledb/Join.java代码实现如下：
package simpledb;
import java.util.*;
/**
* The Join operator implements the relational join operation.
*/
public class Join extends Operator {
private static final long serialVersionUID = 1L;
private JoinPredicate p;
private OpIterator child1;
private OpIterator child2;
private Tuple t;
/**
* Constructor. Accepts two children to join and the predicate to join them
* on
*
* @param p
* The predicate to use to join the children
* @param child1
* Iterator for the left(outer) relation to join
* @param child2
* Iterator for the right(inner) relation to join
*/
public Join(JoinPredicate p, OpIterator child1, OpIterator child2) {
// some code goes here
this.p = p;
this.child1 = child1;
this.child2 = child2;
t = null;
}
public JoinPredicate getJoinPredicate() {
// some code goes here
return p;
}
/**
* @return
* the field name of join field1. Should be quantified by
* alias or table name.
* */
public String getJoinField1Name() {
// some code goes here
return child1.getTupleDesc().getFieldName(p.getField1());
}
/**
* @return
* the field name of join field2. Should be quantified by
* alias or table name.
* */
public String getJoinField2Name() {
// some code goes here
return child2.getTupleDesc().getFieldName(p.getField2());
}
/**
* @see simpledb.TupleDesc#merge(TupleDesc, TupleDesc) for possible
* implementation logic.
*/
public TupleDesc getTupleDesc() {
// some code goes here
return TupleDesc.merge(child1.getTupleDesc(),child2.getTupleDesc());
}
public void open() throws DbException, NoSuchElementException,
TransactionAbortedException {
// some code goes here
child1.open();
child2.open();
super.open();
}
public void close() {
// some code goes here
super.close();
child2.close();
child1.close();
}
public void rewind() throws DbException, TransactionAbortedException {
// some code goes here
child1.rewind();
child2.rewind();
}
/**
* Returns the next tuple generated by the join, or null if there are no
* more tuples. Logically, this is the next tuple in r1 cross r2 that
* satisfies the join predicate. There are many possible implementations;
* the simplest is a nested loops join.
* <p>
* Note that the tuples returned from this particular implementation of Join
* are simply the concatenation of joining tuples from the left and right
* relation. Therefore, if an equality predicate is used there will be two
* copies of the join attribute in the results. (Removing such duplicate
* columns can be done with an additional projection operator if needed.)
* <p>
* For example, if one tuple is {1,2,3} and the other tuple is {1,5,6},
* joined on equality of the first column, then this returns {1,2,3,1,5,6}.
*
* @return The next matching tuple.
* @see JoinPredicate#filter
*/
protected Tuple fetchNext() throws TransactionAbortedException, DbException {
// some code goes here
while(this.child1.hasNext() || this.t != null){
if(this.child1.hasNext() && this.t == null){
t = child1.next();
}
while(child2.hasNext()){
Tuple t2 = child2.next();
if(p.filter(t,t2)){
TupleDesc td1 = t.getTupleDesc();
TupleDesc td2 = t2.getTupleDesc();
TupleDesc newTd = TupleDesc.merge(td1,td2);
Tuple newTuple = new Tuple(newTd);
newTuple.setRecordId(t.getRecordId());
int i=0;
for(;i<td1.numFields();++i)
newTuple.setField(i,t.getField(i));
for(int j=0;j<td2.numFields();++j)
newTuple.setField(i+j,t2.getField(j));
if(!child2.hasNext()){
child2.rewind();
t = null;
}
return newTuple;
}
}
child2.rewind();
t = null;
}
return null;
}
@Override
public OpIterator[] getChildren() {
// some code goes here
return new OpIterator[] {child1,child2};
}
@Override
public void setChildren(OpIterator[] children) {
// some code goes here
child1 = children[0];
child2 = children[1];
}
}
ant runtest -Dtest=PredicateTest 和 ant runtest -Dtest=JoinPredicateTest 和 ant runtest -Dtest=FilterTest 和 ant runtest -Dtest=JoinTest这些单元测试都BUILD SUCCESSFUL。
ant runsystest -Dtest=FilterTest和ant runsystest -Dtest=JoinTest这些系统测试都BUILD SUCCESSFUL。
2.2. Aggregates
还有一个SimpleDB操作符通过GROUP BY语句实现了基础的SQL aggregates(分组聚合)。现在应该实现五中SQL聚合(COUNT,SUM,AVG,MIN,MAX)以及支持分组，只需要实现对一列的聚合，以及对一列的分组。
为了计算聚合，我们使用Aggregator接口，该接口将新tuple融进存在的一个聚合。Aggregator告诉了用于聚合的运算符，之后，用户代码能够对每一个tuple调用Aggregator.mergeTupleIntoGroup() ，当所有的tuple都被合并后，用户利用运算符得到聚合结果。每个tuple在结果中的格式是(groupValue, aggregateValue)，除非group by field的值为Aggregator.NO_GROUPING，结果的的格式会变成(aggregateValue)。
Exercise 2.
Implement the skeleton methods in:
- src/simpledb/IntegerAggregator.java
- src/simpledb/StringAggregator.java
- src/simpledb/Aggregate.java
At this point, your code should pass the unit tests IntegerAggregatorTest, StringAggregatorTest, and AggregateTest. Furthermore, you should be able to pass the AggregateTest system test.
src/simpledb/IntegerAggregator.java的代码如下：
package simpledb;
import java.util.*;
/**
* Knows how to compute some aggregate over a set of IntFields.
*/
public class IntegerAggregator implements Aggregator {
private static final long serialVersionUID = 1L;
private int gbfield;
private Type gbfieldtype;
private int afield;
private Op what;
// running SUM,MIN,MAX,COUNT
private Map<Field,Integer> groupMap;
private Map<Field,Integer> countMap;
private Map<Field,List<Integer>> avgMap;
/**
* Aggregate constructor
*
* @param gbfield
* the 0-based index of the group-by field in the tuple, or
* NO_GROUPING if there is no grouping
* @param gbfieldtype
* the type of the group by field (e.g., Type.INT_TYPE), or null
* if there is no grouping
* @param afield
* the 0-based index of the aggregate field in the tuple
* @param what
* the aggregation operator
*/
public IntegerAggregator(int gbfield, Type gbfieldtype, int afield, Op what) {
// some code goes here
this.gbfield = gbfield;
this.gbfieldtype = gbfieldtype;
this.afield = afield;
this.what = what;
this.groupMap = new HashMap<>();
this.avgMap = new HashMap<>();
this.countMap = new HashMap<>();
}
/**
* Merge a new tuple into the aggregate, grouping as indicated in the
* constructor
*
* @param tup
* the Tuple containing an aggregate field and a group-by field
*/
public void mergeTupleIntoGroup(Tuple tup) {
// some code goes here
IntField afield = (IntField)tup.getField(this.afield);
Field gbfield = this.gbfield == NO_GROUPING ? null : tup.getField(this.gbfield);
int newValue = afield.getValue();
if(gbfield != null && gbfield.getType()!=this.gbfieldtype){
throw new IllegalArgumentException("Given tuple has wrong type");
}
// get number
switch(this.what){
case MIN:
if(!this.groupMap.containsKey(gbfield))
this.groupMap.put(gbfield,newValue);
else
this.groupMap.put(gbfield,Math.min(this.groupMap.get(gbfield),newValue));
break;
case MAX:
if (!this.groupMap.containsKey(gbfield))
this.groupMap.put(gbfield, newValue);
else
this.groupMap.put(gbfield, Math.max(this.groupMap.get(gbfield), newValue));
break;
case SUM:
if (!this.groupMap.containsKey(gbfield))
this.groupMap.put(gbfield, newValue);
else
this.groupMap.put(gbfield, this.groupMap.get(gbfield) + newValue);
break;
case COUNT:
if (!this.groupMap.containsKey(gbfield))
this.groupMap.put(gbfield, 1);
else
this.groupMap.put(gbfield, this.groupMap.get(gbfield) + 1);
break;
case SC_AVG:
IntField countField = null;
if (gbfield == null)
countField = (IntField)tup.getField(1);
else
countField = (IntField)tup.getField(2);
int countValue = countField.getValue();
if (!this.groupMap.containsKey(gbfield)) {
this.groupMap.put(gbfield, newValue);
this.countMap.put(gbfield, countValue);
} else {
this.groupMap.put(gbfield, this.groupMap.get(gbfield) + newValue);
this.countMap.put(gbfield, this.countMap.get(gbfield) + countValue);
}
case SUM_COUNT:
case AVG:
if (!this.avgMap.containsKey(gbfield)) {
List<Integer> l = new ArrayList<>();
l.add(newValue);
this.avgMap.put(gbfield, l);
} else {
// reference
List<Integer> l = this.avgMap.get(gbfield);
l.add(newValue);
}
break;
default:
throw new IllegalArgumentException("Aggregate not supported!");
}
}
/**
* Create a OpIterator over group aggregate results.
*
* @return a OpIterator whose tuples are the pair (groupVal, aggregateVal)
* if using group, or a single (aggregateVal) if no grouping. The
* aggregateVal is determined by the type of aggregate specified in
* the constructor.
*/
public OpIterator iterator() {
// some code goes here
return new IntAggIterator();
}
private class IntAggIterator extends AggregateIterator {
private Iterator<Map.Entry<Field, List<Integer>>> avgIt;
private boolean isAvg;
private boolean isSCAvg;
private boolean isSumCount;
IntAggIterator() {
super(groupMap, gbfieldtype);
this.isAvg = what.equals(Op.AVG);
this.isSCAvg = what.equals(Op.SC_AVG);
this.isSumCount = what.equals(Op.SUM_COUNT);
if (isSumCount) {
this.td = new TupleDesc(new Type[] {this.itgbfieldtype, Type.INT_TYPE, Type.INT_TYPE},
new String[] {"groupVal", "sumVal", "countVal"});
}
}
@Override
public void open() throws DbException, TransactionAbortedException {
super.open();
if (this.isAvg || this.isSumCount)
this.avgIt = avgMap.entrySet().iterator();
else {
this.avgIt = null;
}
}
@Override
public boolean hasNext() throws DbException, TransactionAbortedException {
if (this.isAvg || this.isSumCount)
return avgIt.hasNext();
return super.hasNext();
}
@Override
public Tuple next() throws DbException, TransactionAbortedException, NoSuchElementException {
Tuple rtn = new Tuple(td);
if (this.isAvg || this.isSumCount) {
Map.Entry<Field, List<Integer>> avgOrSumCountEntry = this.avgIt.next();
Field avgOrSumCountField = avgOrSumCountEntry.getKey();
List<Integer> avgOrSumCountList = avgOrSumCountEntry.getValue();
if (this.isAvg) {
int value = this.sumList(avgOrSumCountList) / avgOrSumCountList.size();
this.setFields(rtn, value, avgOrSumCountField);
return rtn;
} else {
this.setFields(rtn, sumList(avgOrSumCountList), avgOrSumCountField);
if (avgOrSumCountField != null)
rtn.setField(2, new IntField(avgOrSumCountList.size()));
else
rtn.setField(1, new IntField(avgOrSumCountList.size()));
return rtn;
}
} else if (this.isSCAvg) {
Map.Entry<Field, Integer> entry = this.it.next();
Field f = entry.getKey();
this.setFields(rtn, entry.getValue() / countMap.get(f), f);
return rtn;
}
return super.next();
}
@Override
public void rewind() throws DbException, TransactionAbortedException {
super.rewind();
if (this.isAvg || this.isSumCount)
this.avgIt = avgMap.entrySet().iterator();
}
@Override
public TupleDesc getTupleDesc() {
return super.getTupleDesc();
}
@Override
public void close() {
super.close();
this.avgIt = null;
}
private int sumList(List<Integer> l) {
int sum = 0;
for (int i : l)
sum += i;
return sum;
}
}
}
src/simpledb/StringAggregator.java的代码如下：
package simpledb;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;
import java.util.NoSuchElementException;
/**
* Knows how to compute some aggregate over a set of StringFields.
*/
public class StringAggregator implements Aggregator {
private static final long serialVersionUID = 1L;
private int gbfield;
private Type gbfieldtype;
private int afield;
private Op what;
private Map<Field, Integer> groupMap;
/**
* Aggregate constructor
* @param gbfield the 0-based index of the group-by field in the tuple, or NO_GROUPING if there is no grouping
* @param gbfieldtype the type of the group by field (e.g., Type.INT_TYPE), or null if there is no grouping
* @param afield the 0-based index of the aggregate field in the tuple
* @param what aggregation operator to use -- only supports COUNT
* @throws IllegalArgumentException if what != COUNT
*/
public StringAggregator(int gbfield, Type gbfieldtype, int afield, Op what) {
if (!what.equals(Op.COUNT))
throw new IllegalArgumentException("Only COUNT is supported for String fields!");
this.gbfield = gbfield;
this.gbfieldtype = gbfieldtype;
this.afield = afield;
this.what = what;
this.groupMap = new HashMap<>();
}
/**
* Merge a new tuple into the aggregate, grouping as indicated in the constructor
* @param tup the Tuple containing an aggregate field and a group-by field
*/
public void mergeTupleIntoGroup(Tuple tup) {
// some code goes here
StringField afield = (StringField) tup.getField(this.afield);
Field gbfield = this.gbfield == NO_GROUPING ? null : tup.getField(this.gbfield);
String newValue = afield.getValue();
if (gbfield != null && gbfield.getType() != this.gbfieldtype) {
throw new IllegalArgumentException("Given tuple has wrong type");
}
if (!this.groupMap.containsKey(gbfield))
this.groupMap.put(gbfield, 1);
else
this.groupMap.put(gbfield, this.groupMap.get(gbfield) + 1);
}
/**
* Create a OpIterator over group aggregate results.
*
* @return a OpIterator whose tuples are the pair (groupVal,
* aggregateVal) if using group, or a single (aggregateVal) if no
* grouping. The aggregateVal is determined by the type of
* aggregate specified in the constructor.
*/
public OpIterator iterator() {
return new AggregateIterator(this.groupMap, this.gbfieldtype);
}
}
class AggregateIterator implements OpIterator {
protected Iterator<Map.Entry<Field, Integer>> it;
TupleDesc td;
private Map<Field, Integer> groupMap;
protected Type itgbfieldtype;
public AggregateIterator(Map<Field, Integer> groupMap, Type gbfieldtype) {
this.groupMap = groupMap;
this.itgbfieldtype = gbfieldtype;
// no grouping
if (this.itgbfieldtype == null)
this.td = new TupleDesc(new Type[] {Type.INT_TYPE}, new String[] {"aggregateVal"});
else
this.td = new TupleDesc(new Type[] {this.itgbfieldtype, Type.INT_TYPE}, new String[] {"groupVal", "aggregateVal"});
}
@Override
public void open() throws DbException, TransactionAbortedException {
this.it = groupMap.entrySet().iterator();
}
@Override
public boolean hasNext() throws DbException, TransactionAbortedException {
return it.hasNext();
}
@Override
public Tuple next() throws DbException, TransactionAbortedException, NoSuchElementException {
Map.Entry<Field, Integer> entry = this.it.next();
Field f = entry.getKey();
Tuple rtn = new Tuple(this.td);
this.setFields(rtn, entry.getValue(), f);
return rtn;
}
@Override
public void rewind() throws DbException, TransactionAbortedException {
this.it = groupMap.entrySet().iterator();
}
@Override
public TupleDesc getTupleDesc() {
return this.td;
}
@Override
public void close() {
this.it = null;
this.td = null;
}
void setFields(Tuple rtn, int value, Field f) {
if (f == null) {
rtn.setField(0, new IntField(value));
} else {
rtn.setField(0, f);
rtn.setField(1, new IntField(value));
}
}
}src/simpledb/Aggregate.java的代码如下：
package simpledb;
import java.util.*;
/**
* The Aggregation operator that computes an aggregate (e.g., sum, avg, max,
* min). Note that we only support aggregates over a single column, grouped by a
* single column.
*/
public class Aggregate extends Operator {
private static final long serialVersionUID = 1L;
private OpIterator child;
private int afield;
private int gfield;
private Aggregator.Op aop;
private Aggregator aggregator;
private OpIterator it;
private TupleDesc td;
/**
* Constructor.
*
* Implementation hint: depending on the type of afield, you will want to
* construct an {@link IntegerAggregator} or {@link StringAggregator} to help
* you with your implementation of readNext().
*
*
* @param child
* The OpIterator that is feeding us tuples.
* @param afield
* The column over which we are computing an aggregate.
* @param gfield
* The column over which we are grouping the result, or -1 if
* there is no grouping
* @param aop
* The aggregation operator to use
*/
public Aggregate(OpIterator child, int afield, int gfield, Aggregator.Op aop) {
// some code goes here
this.child = child;
this.afield = afield;
this.gfield = gfield;
this.aop = aop;
Type gfieldtype = gfield == -1 ? null : this.child.getTupleDesc().getFieldType(this.gfield);
if(this.child.getTupleDesc().getFieldType(this.afield) == (Type.STRING_TYPE)){
this.aggregator = new StringAggregator(this.gfield,gfieldtype,this.afield,this.aop);
}else{
this.aggregator = new IntegerAggregator(this.gfield,gfieldtype,this.afield,this.aop);
}
this.it = this.aggregator.iterator();
// create tupleDesc for agg
List<Type> types = new ArrayList<>();
List<String> names = new ArrayList<>();
// group field
if (gfieldtype != null) {
types.add(gfieldtype);
names.add(this.child.getTupleDesc().getFieldName(this.gfield));
}
types.add(this.child.getTupleDesc().getFieldType(this.afield));
names.add(this.child.getTupleDesc().getFieldName(this.afield));
if (aop.equals(Aggregator.Op.SUM_COUNT)) {
types.add(Type.INT_TYPE);
names.add("COUNT");
}
assert (types.size() == names.size());
this.td = new TupleDesc(types.toArray(new Type[types.size()]), names.toArray(new String[names.size()]));
}
/**
* @return If this aggregate is accompanied by a groupby, return the groupby
* field index in the <b>INPUT</b> tuples. If not, return
* {@link simpledb.Aggregator#NO_GROUPING}
* */
public int groupField() {
// some code goes here
return this.gfield;
}
/**
* @return If this aggregate is accompanied by a group by, return the name
* of the groupby field in the <b>OUTPUT</b> tuples. If not, return
* null;
* */
public String groupFieldName() {
// some code goes here
return this.td.getFieldName(0);
}
/**
* @return the aggregate field
* */
public int aggregateField() {
// some code goes here
return this.afield;
}
/**
* @return return the name of the aggregate field in the <b>OUTPUT</b>
* tuples
* */
public String aggregateFieldName() {
// some code goes here
if(this.gfield == -1)
return this.td.getFieldName(0);
else
return this.td.getFieldName(1);
}
/**
* @return return the aggregate operator
* */
public Aggregator.Op aggregateOp() {
// some code goes here
return this.aop;
}
public static String nameOfAggregatorOp(Aggregator.Op aop) {
return aop.toString();
}
public void open() throws NoSuchElementException, DbException,
TransactionAbortedException {
// some code goes here
this.child.open();
while (this.child.hasNext())
this.aggregator.mergeTupleIntoGroup(this.child.next());
this.it.open();
super.open();
}
/**
* Returns the next tuple. If there is a group by field, then the first
* field is the field by which we are grouping, and the second field is the
* result of computing the aggregate. If there is no group by field, then
* the result tuple should contain one field representing the result of the
* aggregate. Should return null if there are no more tuples.
*/
protected Tuple fetchNext() throws TransactionAbortedException, DbException {
// some code goes here
while (this.it.hasNext())
return this.it.next();
return null;
}
public void rewind() throws DbException, TransactionAbortedException {
// some code goes here
this.child.rewind();
this.it.rewind();
}
/**
* Returns the TupleDesc of this Aggregate. If there is no group by field,
* this will have one field - the aggregate column. If there is a group by
* field, the first field will be the group by field, and the second will be
* the aggregate value column.
*
* The name of an aggregate column should be informative. For example:
* "aggName(aop) (child_td.getFieldName(afield))" where aop and afield are
* given in the constructor, and child_td is the TupleDesc of the child
* iterator.
*/
public TupleDesc getTupleDesc() {
// some code goes here
return this.td;
}
public void close() {
// some code goes here
super.close();
this.child.close();
this.it.close();
}
@Override
public OpIterator[] getChildren() {
// some code goes here
return new OpIterator[] {this.child};
}
@Override
public void setChildren(OpIterator[] children) {
// some code goes here
this.child = children[0];
List<Type> types = new ArrayList<>();
List<String> names = new ArrayList<>();
Type gfieldtype = gfield == -1 ? null : this.child.getTupleDesc().getFieldType(this.gfield);
// group field
if (gfieldtype != null) {
types.add(gfieldtype);
names.add(this.child.getTupleDesc().getFieldName(this.gfield));
}
types.add(this.child.getTupleDesc().getFieldType(this.afield));
names.add(this.child.getTupleDesc().getFieldName(this.afield));
if (aop.equals(Aggregator.Op.SUM_COUNT)) {
types.add(Type.INT_TYPE);
names.add("COUNT");
}
assert (types.size() == names.size());
this.td = new TupleDesc(types.toArray(new Type[types.size()]), names.toArray(new String[names.size()]));
}
}
ant runtest -Dtest=IntegerAggregatorTest 和 ant runtest -Dtest=StringAggregatorTest 和 ant runtest -Dtest=FilterTest 和 ant runtest -Dtest=AggregateTest这些单元测试都BUILD SUCCESSFUL。
ant runsystest -Dtest=AggregateTest系统测试BUILD SUCCESSFUL。
2.3. HeapFile Mutability
现在，需要实现方法支持修改表。我们先从独立物理页和文件的层次开始，这有两组主要的操作符，adding tuples和removing tuples。
Removing tuples: 要移除一个Tuple，需要实现deleteTuple。Tuples包含了RecordIDs，能帮助找到tuple存储的物理页，所以大致思路就是找到tuple所属的物理页，正确地修改物理页的headers。
Adding tuples: HeapFile.java中的insertTuple方法负责添加一个Tuple到heap file。为了增加一个新的tuple到HeapFile，需要找到页中一个空闲的slot。如果HeapFile中不存在这样的页，需要创建一个新页，并添加新页到磁盘上的物理文件中。需要保证tuple中的RecordID正确地更新。
Exercise 3.
Implement the remaining skeleton methods in:
- src/simpledb/HeapPage.java
- src/simpledb/HeapFile.java
(Note that you do not necessarily need to implement writePage at this point).
To implement HeapPage, you will need to modify the header bitmap for methods such as insertTuple() and deleteTuple(). You may find that the getNumEmptySlots() and isSlotUsed() methods we asked you to implement in Lab 1 serve as useful abstractions. Note that there is a markSlotUsed method provided as an abstraction to modify the filled or cleared status of a tuple in the page header.
Note that it is important that the HeapFile.insertTuple() and HeapFile.deleteTuple() methods access pages using the BufferPool.getPage() method; otherwise, your implementation of transactions in the next lab will not work properly.
Implement the following skeleton methods in src/simpledb/BufferPool.java:
- insertTuple()
- deleteTuple()
These methods should call the appropriate methods in the HeapFile that belong to the table being modified (this extra level of indirection is needed to support other types of files — like indices — in the future).
At this point, your code should pass the unit tests in HeapPageWriteTest and HeapFileWriteTest, as well as BufferPoolWriteTest.
src/simpledb/HeapPage.java 的代码如下：
package simpledb;
import java.util.*;
import java.io.*;
/**
* Each instance of HeapPage stores data for one page of HeapFiles and
* implements the Page interface that is used by BufferPool.
*
* @see HeapFile
* @see BufferPool
*
*/
public class HeapPage implements Page {
final HeapPageId pid;
final TupleDesc td;
final byte header[];
final Tuple tuples[];
final int numSlots;
byte[] oldData;
private final Byte oldDataLock=new Byte((byte)0);
// the transaction id which changed the page to dirty
private TransactionId dirtyId;
// if the page is dirty
private boolean dirty;
/**
* Create a HeapPage from a set of bytes of data read from disk.
* The format of a HeapPage is a set of header bytes indicating
* the slots of the page that are in use, some number of tuple slots.
* Specifically, the number of tuples is equal to: <p>
* floor((BufferPool.getPageSize()*8) / (tuple size * 8 + 1))
* <p> where tuple size is the size of tuples in this
* database table, which can be determined via {@link Catalog#getTupleDesc}.
* The number of 8-bit header words is equal to:
* <p>
* ceiling(no. tuple slots / 8)
* <p>
* @see Database#getCatalog
* @see Catalog#getTupleDesc
* @see BufferPool#getPageSize()
*/
public HeapPage(HeapPageId id, byte[] data) throws IOException {
this.pid = id;
this.td = Database.getCatalog().getTupleDesc(id.getTableId());
this.numSlots = getNumTuples();
DataInputStream dis = new DataInputStream(new ByteArrayInputStream(data));
// allocate and read the header slots of this page
header = new byte[getHeaderSize()];
for (int i=0; i<header.length; i++)
header[i] = dis.readByte();
tuples = new Tuple[numSlots];
try{
// allocate and read the actual records of this page
for (int i=0; i<tuples.length; i++)
tuples[i] = readNextTuple(dis,i);
}catch(NoSuchElementException e){
e.printStackTrace();
}
dis.close();
setBeforeImage();
}
/** Retrieve the number of tuples on this page.
@return the number of tuples on this page
*/
private int getNumTuples() {
// some code goes here
//_tuples per page_ = floor((_page size_ * 8) / (_tuple size_ * 8 + 1))
int num = (int)Math.floor((BufferPool.getPageSize()*8*1.0)/(td.getSize()*8+1));
return num;
}
/**
* Computes the number of bytes in the header of a page in a HeapFile with each tuple occupying tupleSize bytes
* @return the number of bytes in the header of a page in a HeapFile with each tuple occupying tupleSize bytes
*/
private int getHeaderSize() {
// some code goes here
// headerBytes = ceiling(tupsPerPage/8)
return (int)Math.ceil(getNumTuples()*1.0/8);
}
/** Return a view of this page before it was modified
-- used by recovery */
public HeapPage getBeforeImage(){
try {
byte[] oldDataRef = null;
synchronized(oldDataLock)
{
oldDataRef = oldData;
}
return new HeapPage(pid,oldDataRef);
} catch (IOException e) {
e.printStackTrace();
//should never happen -- we parsed it OK before!
System.exit(1);
}
return null;
}
public void setBeforeImage() {
synchronized(oldDataLock)
{
oldData = getPageData().clone();
}
}
/**
* @return the PageId associated with this page.
*/
public HeapPageId getId() {
// some code goes here
return pid;
}
/**
* Suck up tuples from the source file.
*/
private Tuple readNextTuple(DataInputStream dis, int slotId) throws NoSuchElementException {
// if associated bit is not set, read forward to the next tuple, and
// return null.
if (!isSlotUsed(slotId)) {
for (int i=0; i<td.getSize(); i++) {
try {
dis.readByte();
} catch (IOException e) {
throw new NoSuchElementException("error reading empty tuple");
}
}
return null;
}
// read fields in the tuple
Tuple t = new Tuple(td);
RecordId rid = new RecordId(pid, slotId);
t.setRecordId(rid);
try {
for (int j=0; j<td.numFields(); j++) {
Field f = td.getFieldType(j).parse(dis);
t.setField(j, f);
}
} catch (java.text.ParseException e) {
e.printStackTrace();
throw new NoSuchElementException("parsing error!");
}
return t;
}
/**
* Generates a byte array representing the contents of this page.
* Used to serialize this page to disk.
* <p>
* The invariant here is that it should be possible to pass the byte
* array generated by getPageData to the HeapPage constructor and
* have it produce an identical HeapPage object.
*
* @see #HeapPage
* @return A byte array correspond to the bytes of this page.
*/
public byte[] getPageData() {
int len = BufferPool.getPageSize();
ByteArrayOutputStream baos = new ByteArrayOutputStream(len);
DataOutputStream dos = new DataOutputStream(baos);
// create the header of the page
for (int i=0; i<header.length; i++) {
try {
dos.writeByte(header[i]);
} catch (IOException e) {
// this really shouldn't happen
e.printStackTrace();
}
}
// create the tuples
for (int i=0; i<tuples.length; i++) {
// empty slot
if (!isSlotUsed(i)) {
for (int j=0; j<td.getSize(); j++) {
try {
dos.writeByte(0);
} catch (IOException e) {
e.printStackTrace();
}
}
continue;
}
// non-empty slot
for (int j=0; j<td.numFields(); j++) {
Field f = tuples[i].getField(j);
try {
f.serialize(dos);
} catch (IOException e) {
e.printStackTrace();
}
}
}
// padding
int zerolen = BufferPool.getPageSize() - (header.length + td.getSize() * tuples.length); //- numSlots * td.getSize();
byte[] zeroes = new byte[zerolen];
try {
dos.write(zeroes, 0, zerolen);
} catch (IOException e) {
e.printStackTrace();
}
try {
dos.flush();
} catch (IOException e) {
e.printStackTrace();
}
return baos.toByteArray();
}
/**
* Static method to generate a byte array corresponding to an empty
* HeapPage.
* Used to add new, empty pages to the file. Passing the results of
* this method to the HeapPage constructor will create a HeapPage with
* no valid tuples in it.
*
* @return The returned ByteArray.
*/
public static byte[] createEmptyPageData() {
int len = BufferPool.getPageSize();
return new byte[len]; //all 0
}
/**
* Delete the specified tuple from the page; the corresponding header bit should be updated to reflect
* that it is no longer stored on any page.
* @throws DbException if this tuple is not on this page, or tuple slot is
* already empty.
* @param t The tuple to delete
*/
public void deleteTuple(Tuple t) throws DbException {
// some code goes here
// not necessary for lab1
int tid = t.getRecordId().getTupleNumber();
if(tuples[tid] == null){
throw new DbException("tuple does not exist");
}
if(!isSlotUsed(tid)){
throw new DbException("the slot is already empty");
}
// else if(!t.equals(tuples[tid])){
// throw new DbException(String.format("tuple does not exits %d and %d",t.getRecordId().hashCode(),tuples[tid].getRecordId().hashCode()));
// }
else{
markSlotUsed(tid,false);
tuples[tid] = null;
}
}
/**
* Adds the specified tuple to the page; the tuple should be updated to reflect
* that it is now stored on this page.
* @throws DbException if the page is full (no empty slots) or tupledesc
* is mismatch.
* @param t The tuple to add.
*/
public void insertTuple(Tuple t) throws DbException {
// some code goes here
// not necessary for lab1
if(getNumEmptySlots() == 0 || !t.getTupleDesc().equals(td)){
throw new DbException("page is full or tuple descriptor does not match");
}
for(int i=0;i<numSlots;++i){
if(!isSlotUsed(i)){
markSlotUsed(i,true);
t.setRecordId(new RecordId(pid,i));
tuples[i] = t;
break;
}
}
}
/**
* Marks this page as dirty/not dirty and record that transaction
* that did the dirtying
*/
public void markDirty(boolean dirty, TransactionId tid) {
// some code goes here
// not necessary for lab1
this.dirty = dirty;
this.dirtyId = tid;
}
/**
* Returns the tid of the transaction that last dirtied this page, or null if the page is not dirty
*/
public TransactionId isDirty() {
// some code goes here
// Not necessary for lab1
return this.dirty ? this.dirtyId:null;
}
/**
* Returns the number of empty slots on this page.
*/
public int getNumEmptySlots() {
// some code goes here
int cnt = 0;
for(int i=0;i<numSlots;++i){
if(!isSlotUsed(i)){
++cnt;
}
}
return cnt;
}
/**
* Returns true if associated slot on this page is filled.
*/
public boolean isSlotUsed(int i) {
// some code goes here
// use bitmap
byte b = header[Math.floorDiv(i,8)];
byte mask = (byte)(1<<(i%8));
return (byte)(b&mask) != 0;
}
/**
* Abstraction to fill or clear a slot on this page.
*/
private void markSlotUsed(int i, boolean value) {
// some code goes here
// not necessary for lab1
byte b = header[Math.floorDiv(i,8)];
byte mask = (byte)(1<<(i%8));
if(value){
header[Math.floorDiv(i,8)] = (byte) (b|mask);
}else{
header[Math.floorDiv(i,8)] = (byte) (b&(~mask));
}
}
/**
* @return an iterator over all tuples on this page (calling remove on this iterator throws an UnsupportedOperationException)
* (note that this iterator shouldn't return tuples in empty slots!)
*/
public Iterator<Tuple> iterator() {
// some code goes here
ArrayList<Tuple> filledTuples = new ArrayList<Tuple>();
for(int i=0;i<numSlots;++i){
if(isSlotUsed(i)){
filledTuples.add(tuples[i]);
}
}
return filledTuples.iterator();
}
}
src/simpledb/HeapFile.java的代码如下：
package simpledb;
import javax.xml.crypto.Data;
import java.io.*;
import java.util.*;
/**
* HeapFile is an implementation of a DbFile that stores a collection of tuples
* in no particular order. Tuples are stored on pages, each of which is a fixed
* size, and the file is simply a collection of those pages. HeapFile works
* closely with HeapPage. The format of HeapPages is described in the HeapPage
* constructor.
*
* @see simpledb.HeapPage#HeapPage
* @author Sam Madden
*/
public class HeapFile implements DbFile {
private final File file;
private final TupleDesc td;
/**
* Constructs a heap file backed by the specified file.
*
* @param f
* the file that stores the on-disk backing store for this heap
* file.
*/
public HeapFile(File f, TupleDesc td) {
// some code goes here
this.file = f;
this.td = td;
}
/**
* Returns the File backing this HeapFile on disk.
*
* @return the File backing this HeapFile on disk.
*/
public File getFile() {
// some code goes here
return file;
}
/**
* Returns an ID uniquely identifying this HeapFile. Implementation note:
* you will need to generate this tableid somewhere to ensure that each
* HeapFile has a "unique id," and that you always return the same value for
* a particular HeapFile. We suggest hashing the absolute file name of the
* file underlying the heapfile, i.e. f.getAbsoluteFile().hashCode().
*
* @return an ID uniquely identifying this HeapFile.
*/
public int getId() {
// some code goes here
return file.getAbsoluteFile().hashCode();
}
/**
* Returns the TupleDesc of the table stored in this DbFile.
*
* @return TupleDesc of this DbFile.
*/
public TupleDesc getTupleDesc() {
// some code goes here
return td;
}
// see DbFile.java for javadocs
public Page readPage(PageId pid) {
// some code goes here
int tableId = pid.getTableId();
int pgNo = pid.getPageNumber();
RandomAccessFile f = null;
try{
f = new RandomAccessFile(file,"r");
if((pgNo+1)*BufferPool.getPageSize() > f.length()){
f.close();
throw new IllegalArgumentException(String.format("table %d page %d is invalid", tableId, pgNo));
}
byte[] bytes = new byte[BufferPool.getPageSize()];
f.seek(pgNo * BufferPool.getPageSize());
// big end
int read = f.read(bytes,0,BufferPool.getPageSize());
if(read != BufferPool.getPageSize()){
throw new IllegalArgumentException(String.format("table %d page %d read %d bytes", tableId, pgNo, read));
}
HeapPageId id = new HeapPageId(pid.getTableId(),pid.getPageNumber());
return new HeapPage(id,bytes);
}catch (IOException e){
e.printStackTrace();
}finally {
try{
f.close();
}catch (Exception e){
e.printStackTrace();
}
}
throw new IllegalArgumentException(String.format("table %d page %d is invalid", tableId, pgNo));
}
// see DbFile.java for javadocs
public void writePage(Page page) throws IOException {
// some code goes here
// not necessary for lab1
int pgNo = page.getId().getPageNumber();
if(pgNo > numPages()){
throw new IllegalArgumentException();
}
int pgSize = BufferPool.getPageSize();
//write IO
RandomAccessFile f = new RandomAccessFile(file,"rw");
// set offset
f.seek(pgNo*pgSize);
// write
byte[] data = page.getPageData();
f.write(data);
f.close();
}
/**
* Returns the number of pages in this HeapFile.
*/
public int numPages() {
// some code goes here
int num = (int)Math.floor(file.length()*1.0/BufferPool.getPageSize());
return num;
}
// see DbFile.java for javadocs
public ArrayList<Page> insertTuple(TransactionId tid, Tuple t)
throws DbException, IOException, TransactionAbortedException {
// some code goes here
ArrayList<Page> pageList= new ArrayList<Page>();
for(int i=0;i<numPages();++i){
// took care of getting new page
HeapPage p = (HeapPage) Database.getBufferPool().getPage(tid,
new HeapPageId(this.getId(),i),Permissions.READ_WRITE);
if(p.getNumEmptySlots() == 0)
continue;
p.insertTuple(t);
pageList.add(p);
return pageList;
}
// no new page
BufferedOutputStream bw = new BufferedOutputStream(new FileOutputStream(file,true));
byte[] emptyData = HeapPage.createEmptyPageData();
bw.write(emptyData);
bw.close();
// load into cache
HeapPage p = (HeapPage) Database.getBufferPool().getPage(tid,
new HeapPageId(getId(),numPages()-1),Permissions.READ_WRITE);
p.insertTuple(t);
pageList.add(p);
return pageList;
}
// see DbFile.java for javadocs
public ArrayList<Page> deleteTuple(TransactionId tid, Tuple t) throws DbException,
TransactionAbortedException {
// some code goes here
ArrayList<Page> pageList = new ArrayList<Page>();
HeapPage p = (HeapPage) Database.getBufferPool().getPage(tid,
t.getRecordId().getPageId(),Permissions.READ_WRITE);
p.deleteTuple(t);
pageList.add(p);
return pageList;
}
// see DbFile.java for javadocs
public DbFileIterator iterator(TransactionId tid) {
// some code goes here
return new HeapFileIterator(this,tid);
}
private static final class HeapFileIterator implements DbFileIterator{
private final HeapFile heapFile;
private final TransactionId tid;
private Iterator<Tuple> it;
private int whichPage;
public HeapFileIterator(HeapFile file,TransactionId tid){
this.heapFile = file;
this.tid = tid;
}
@Override
public void open() throws DbException, TransactionAbortedException {
// TODO Auto-generated method stub
whichPage = 0;
it = getPageTuples(whichPage);
}
private Iterator<Tuple> getPageTuples(int pageNumber) throws TransactionAbortedException, DbException{
if(pageNumber >= 0 && pageNumber < heapFile.numPages()){
HeapPageId pid = new HeapPageId(heapFile.getId(),pageNumber);
HeapPage page = (HeapPage)Database.getBufferPool().getPage(tid, pid, Permissions.READ_ONLY);
return page.iterator();
}else{
throw new DbException(String.format("heapfile %d does not contain page %d!", pageNumber,heapFile.getId()));
}
}
@Override
public boolean hasNext() throws DbException, TransactionAbortedException {
// TODO Auto-generated method stub
if(it == null){
return false;
}
if(!it.hasNext()){
if(whichPage < (heapFile.numPages()-1)){
whichPage++;
it = getPageTuples(whichPage);
return it.hasNext();
}else{
return false;
}
}else{
return true;
}
}
@Override
public Tuple next() throws DbException, TransactionAbortedException, NoSuchElementException {
// TODO Auto-generated method stub
if(it == null || !it.hasNext()){
throw new NoSuchElementException();
}
return it.next();
}
@Override
public void rewind() throws DbException, TransactionAbortedException {
// TODO Auto-generated method stub
close();
open();
}
@Override
public void close() {
// TODO Auto-generated method stub
it = null;
}
}
}
src/simpledb/BufferPool.java的代码如下：
public void insertTuple(TransactionId tid, int tableId, Tuple t)
throws DbException, IOException, TransactionAbortedException {
// some code goes here
// not necessary for lab1
DbFile f = Database.getCatalog().getDatabaseFile(tableId);
updateBufferPool(f.insertTuple(tid,t),tid);
}
/**
* Remove the specified tuple from the buffer pool.
* Will acquire a write lock on the page the tuple is removed from and any
* other pages that are updated. May block if the lock(s) cannot be acquired.
*
* Marks any pages that were dirtied by the operation as dirty by calling
* their markDirty bit, and adds versions of any pages that have
* been dirtied to the cache (replacing any existing versions of those pages) so
* that future requests see up-to-date pages.
*
* @param tid the transaction deleting the tuple.
* @param t the tuple to delete
*/
public void deleteTuple(TransactionId tid, Tuple t)
throws DbException, IOException, TransactionAbortedException {
// some code goes here
// not necessary for lab1
DbFile f = Database.getCatalog().getDatabaseFile(t.getRecordId().getPageId().getTableId());
updateBufferPool(f.deleteTuple(tid,t),tid);
}
private void updateBufferPool(ArrayList<Page> pagelist,TransactionId tid) throws DbException{
for(Page p:pagelist){
p.markDirty(true,tid);
// update bufferpool
if(pageStore.size() > numPages)
evictPage();
pageStore.put(p.getId(),p);
}
}
ant runtest -Dtest=HeapPageWriteTest 和 ant runtest -Dtest=HeapFileWriteTest 和 ant runtest -Dtest=BufferPoolWriteTest都BUILD SUCCESSFUL。
2.4. Insertion and deletion
现在需要写HeapFile添加和移除tuples的机制，需要实现Insert 和 Delete操作符。
实现insert和delete查询的策略中，最高的操作符是Insert和Delete，用于修改磁盘上的页数据，这些操作符返回受影响的tuples个数。
Insert: 这个操作符添加 child operator读取的tuples 到 tableid 代表的表中，需要用到BufferPool.insertTuple()方法实现。
Delete:这个操作符删除 child operator读取的tuples 到 tableid 代表的表中，需要用到BufferPool.deleteTuple()方法实现。
Exercise 4.
Implement the skeleton methods in:
- src/simpledb/Insert.java
- src/simpledb/Delete.java
At this point, your code should pass the unit tests in InsertTest. We have not provided unit tests for
Delete. Furthermore, you should be able to pass the InsertTest and DeleteTest system tests.
src/simpledb/Insert.java的代码如下：
package simpledb;
import java.io.IOException;
/**
* Inserts tuples read from the child operator into the tableId specified in the
* constructor
*/
public class Insert extends Operator {
private static final long serialVersionUID = 1L;
private TransactionId tid;
private OpIterator child;
private int tableId;
private final TupleDesc td;
// helper for fetchNext
private int counter;
private boolean called;
/**
* Constructor.
*
* @param t
* The transaction running the insert.
* @param child
* The child operator from which to read tuples to be inserted.
* @param tableId
* The table in which to insert tuples.
* @throws DbException
* if TupleDesc of child differs from table into which we are to
* insert.
*/
public Insert(TransactionId t, OpIterator child, int tableId)
throws DbException {
// some code goes here
if(!child.getTupleDesc().equals(Database.getCatalog().getTupleDesc(tableId))){
throw new DbException("TupleDesc does not match!");
}
this.tid = t;
this.child = child;
this.tableId = tableId;
this.td = new TupleDesc(new Type[]{Type.INT_TYPE},new String[]{"number of inserted tuples"});
this.counter = -1;
this.called = false;
}
public TupleDesc getTupleDesc() {
// some code goes here
return this.td;
}
public void open() throws DbException, TransactionAbortedException {
// some code goes here
this.counter = 0;
this.child.open();
super.open();
}
public void close() {
// some code goes here
super.close();
this.child.close();
this.counter = -1;
this.called = false;
}
public void rewind() throws DbException, TransactionAbortedException {
// some code goes here
this.child.rewind();
this.counter = 0;
this.called = false;
}
/**
* Inserts tuples read from child into the tableId specified by the
* constructor. It returns a one field tuple containing the number of
* inserted records. Inserts should be passed through BufferPool. An
* instances of BufferPool is available via Database.getBufferPool(). Note
* that insert DOES NOT need check to see if a particular tuple is a
* duplicate before inserting it.
*
* @return A 1-field tuple containing the number of inserted records, or
* null if called more than once.
* @see Database#getBufferPool
* @see BufferPool#insertTuple
*/
protected Tuple fetchNext() throws TransactionAbortedException, DbException {
// some code goes here
if (this.called)
return null;
this.called = true;
while (this.child.hasNext()) {
Tuple t = this.child.next();
try {
Database.getBufferPool().insertTuple(this.tid, this.tableId, t);
this.counter++;
} catch (IOException e) {
e.printStackTrace();
break;
}
}
Tuple tu = new Tuple(this.td);
tu.setField(0, new IntField(this.counter));
return tu;
}
@Override
public OpIterator[] getChildren() {
// some code goes here
return new OpIterator[] {this.child};
}
@Override
public void setChildren(OpIterator[] children) {
// some code goes here
this.child = children[0];
}
}
src/simpledb/Delete.java的代码如下：
package simpledb;
import java.io.IOException;
/**
* The delete operator. Delete reads tuples from its child operator and removes
* them from the table they belong to.
*/
public class Delete extends Operator {
private static final long serialVersionUID = 1L;
private TransactionId tid;
private OpIterator child;
private final TupleDesc td;
private int counter;
private boolean called;
/**
* Constructor specifying the transaction that this delete belongs to as
* well as the child to read from.
*
* @param t
* The transaction this delete runs in
* @param child
* The child operator from which to read tuples for deletion
*/
public Delete(TransactionId t, OpIterator child) {
// some code goes here
this.tid = t;
this.child = child;
this.td = new TupleDesc(new Type[] {Type.INT_TYPE}, new String[] {"number of deleted tuples"});
this.counter = -1;
this.called = false;
}
public TupleDesc getTupleDesc() {
// some code goes here
return this.td;
}
public void open() throws DbException, TransactionAbortedException {
// some code goes here
this.child.open();
super.open();
this.counter = 0;
}
public void close() {
// some code goes here
super.close();
this.child.close();
this.counter = -1;
}
public void rewind() throws DbException, TransactionAbortedException {
// some code goes here
this.child.rewind();
this.counter = 0;
}
/**
* Deletes tuples as they are read from the child operator. Deletes are
* processed via the buffer pool (which can be accessed via the
* Database.getBufferPool() method.
*
* @return A 1-field tuple containing the number of deleted records.
* @see Database#getBufferPool
* @see BufferPool#deleteTuple
*/
protected Tuple fetchNext() throws TransactionAbortedException, DbException {
// some code goes here
if (this.called)
return null;
this.called = true;
while (this.child.hasNext()) {
Tuple t = this.child.next();
try {
Database.getBufferPool().deleteTuple(this.tid, t);
this.counter++;
} catch (IOException e) {
e.printStackTrace();
break;
}
}
Tuple tu = new Tuple(this.td);
tu.setField(0, new IntField(this.counter));
return tu;
}
@Override
public OpIterator[] getChildren() {
// some code goes here
return new OpIterator[] {this.child};
}
@Override
public void setChildren(OpIterator[] children) {
// some code goes here
this.child = children[0];
}
}
ant runtest -Dtest=InsertTest单元测试BUILD SUCCESSFUL。
ant runsystest -Dtest=InsertTest和ant runsystest -Dtest=DeleteTest系统测试BUILD SUCCESSFUL。
2.5. Page eviction
在lab1中，没有正视buffer pool对于最大页数的限制，现在当页数超出缓存的时候，需要使用淘汰(eviction)策略。
当超出buffer pool定义的numPages时，有一个页应该从pool中被淘汰，让下一个载入，淘汰策略的选择取决于实现者，不一定要非常复杂。
注意到BufferPool 请求实现flushAllPages()这个方法。这个不是buffer pool实际应用中需要的方法，在这里实现只是用于测试，在现实中从不需要调用。
从buffer pool中移除页的唯一方法是evictPage，在evictPage方法中调用flushPage清洗淘汰的dirty page。
Exercise 5.
Fill in the
flushPage()method and additional helper methods to implement page eviction in:
- src/simpledb/BufferPool.java
If you did not implement
writePage()in HeapFile.java above, you will also need to do that here. Finally, you should also implementdiscardPage()to remove a page from the buffer pool without flushing it to disk. We will not testdiscardPage()in this lab, but it will be necessary for future labs.At this point, your code should pass the EvictionTest system test.
Since we will not be checking for any particular eviction policy, this test works by creating a BufferPool with 16 pages (NOTE: while DEFAULT_PAGES is 50, we are initializing the BufferPool with less!), scanning a file with many more than 16 pages, and seeing if the memory usage of the JVM increases by more than 5 MB. If you do not implement an eviction policy correctly, you will not evict enough pages, and will go over the size limitation, thus failing the test.
You have now completed this lab. Good work!
src/simpledb/BufferPool.java代码如下：
package simpledb;
import javax.xml.crypto.Data;
import java.io.*;
import java.util.*;
import java.util.concurrent.ConcurrentHashMap;
/**
* BufferPool manages the reading and writing of pages into memory from
* disk. Access methods call into it to retrieve pages, and it fetches
* pages from the appropriate location.
* <p>
* The BufferPool is also responsible for locking; when a transaction fetches
* a page, BufferPool checks that the transaction has the appropriate
* locks to read/write the page.
*
* @Threadsafe, all fields are final
*/
public class BufferPool {
/** Bytes per page, including header. */
private static final int DEFAULT_PAGE_SIZE = 4096;
private static int pageSize = DEFAULT_PAGE_SIZE;
/** Default number of pages passed to the constructor. This is used by
other classes. BufferPool should use the numPages argument to the
constructor instead. */
public static final int DEFAULT_PAGES = 50;
private final int numPages;
private final ConcurrentHashMap<PageId,Page> pageStore;
/**
* Creates a BufferPool that caches up to numPages pages.
*
* @param numPages maximum number of pages in this buffer pool.
*/
public BufferPool(int numPages) {
// some code goes here
this.numPages = numPages;
pageStore = new ConcurrentHashMap<PageId,Page>();
}
public static int getPageSize() {
return pageSize;
}
// THIS FUNCTION SHOULD ONLY BE USED FOR TESTING!!
public static void setPageSize(int pageSize) {
BufferPool.pageSize = pageSize;
}
// THIS FUNCTION SHOULD ONLY BE USED FOR TESTING!!
public static void resetPageSize() {
BufferPool.pageSize = DEFAULT_PAGE_SIZE;
}
/**
* Retrieve the specified page with the associated permissions.
* Will acquire a lock and may block if that lock is held by another
* transaction.
* <p>
* The retrieved page should be looked up in the buffer pool. If it
* is present, it should be returned. If it is not present, it should
* be added to the buffer pool and returned. If there is insufficient
* space in the buffer pool, a page should be evicted and the new page
* should be added in its place.
*
* @param tid the ID of the transaction requesting the page
* @param pid the ID of the requested page
* @param perm the requested permissions on the page
*/
public Page getPage(TransactionId tid, PageId pid, Permissions perm)
throws TransactionAbortedException, DbException {
// some code goes here
if(!pageStore.containsKey(pid)){
if(pageStore.size()>numPages){
evictPage();
}
DbFile dbfile = Database.getCatalog().getDatabaseFile(pid.getTableId());
Page page = dbfile.readPage(pid);
pageStore.put(pid,page);
}
return pageStore.get(pid);
}
/**
* Releases the lock on a page.
* Calling this is very risky, and may result in wrong behavior. Think hard
* about who needs to call this and why, and why they can run the risk of
* calling it.
*
* @param tid the ID of the transaction requesting the unlock
* @param pid the ID of the page to unlock
*/
public void releasePage(TransactionId tid, PageId pid) {
// some code goes here
// not necessary for lab1|lab2
}
/**
* Release all locks associated with a given transaction.
*
* @param tid the ID of the transaction requesting the unlock
*/
public void transactionComplete(TransactionId tid) throws IOException {
// some code goes here
// not necessary for lab1|lab2
}
/** Return true if the specified transaction has a lock on the specified page */
public boolean holdsLock(TransactionId tid, PageId p) {
// some code goes here
// not necessary for lab1|lab2
return false;
}
/**
* Commit or abort a given transaction; release all locks associated to
* the transaction.
*
* @param tid the ID of the transaction requesting the unlock
* @param commit a flag indicating whether we should commit or abort
*/
public void transactionComplete(TransactionId tid, boolean commit)
throws IOException {
// some code goes here
// not necessary for lab1|lab2
}
/**
* Add a tuple to the specified table on behalf of transaction tid. Will
* acquire a write lock on the page the tuple is added to and any other
* pages that are updated (Lock acquisition is not needed for lab2).
* May block if the lock(s) cannot be acquired.
*
* Marks any pages that were dirtied by the operation as dirty by calling
* their markDirty bit, and adds versions of any pages that have
* been dirtied to the cache (replacing any existing versions of those pages) so
* that future requests see up-to-date pages.
*
* @param tid the transaction adding the tuple
* @param tableId the table to add the tuple to
* @param t the tuple to add
*/
public void insertTuple(TransactionId tid, int tableId, Tuple t)
throws DbException, IOException, TransactionAbortedException {
// some code goes here
// not necessary for lab1
DbFile f = Database.getCatalog().getDatabaseFile(tableId);
updateBufferPool(f.insertTuple(tid,t),tid);
}
/**
* Remove the specified tuple from the buffer pool.
* Will acquire a write lock on the page the tuple is removed from and any
* other pages that are updated. May block if the lock(s) cannot be acquired.
*
* Marks any pages that were dirtied by the operation as dirty by calling
* their markDirty bit, and adds versions of any pages that have
* been dirtied to the cache (replacing any existing versions of those pages) so
* that future requests see up-to-date pages.
*
* @param tid the transaction deleting the tuple.
* @param t the tuple to delete
*/
public void deleteTuple(TransactionId tid, Tuple t)
throws DbException, IOException, TransactionAbortedException {
// some code goes here
// not necessary for lab1
DbFile f = Database.getCatalog().getDatabaseFile(t.getRecordId().getPageId().getTableId());
updateBufferPool(f.deleteTuple(tid,t),tid);
}
private void updateBufferPool(ArrayList<Page> pagelist,TransactionId tid) throws DbException{
for(Page p:pagelist){
p.markDirty(true,tid);
// update bufferpool
if(pageStore.size() > numPages)
evictPage();
pageStore.put(p.getId(),p);
}
}
/**
* Flush all dirty pages to disk.
* NB: Be careful using this routine -- it writes dirty data to disk so will
* break simpledb if running in NO STEAL mode.
*/
public synchronized void flushAllPages() throws IOException {
// some code goes here
// not necessary for lab1
for(Page p:pageStore.values()){
flushPage(p.getId());
}
}
/** Remove the specific page id from the buffer pool.
Needed by the recovery manager to ensure that the
buffer pool doesn't keep a rolled back page in its
cache.
Also used by B+ tree files to ensure that deleted pages
are removed from the cache so they can be reused safely
*/
public synchronized void discardPage(PageId pid) {
// some code goes here
// not necessary for lab1
pageStore.remove(pid);
}
/**
* Flushes a certain page to disk
* @param pid an ID indicating the page to flush
*/
private synchronized void flushPage(PageId pid) throws IOException {
// some code goes here
// not necessary for lab1
Page p = pageStore.get(pid);
TransactionId tid = null;
// flush it if it is dirty
if((tid = p.isDirty())!= null){
Database.getLogFile().logWrite(tid,p.getBeforeImage(),p);
Database.getLogFile().force();
// write to disk
Database.getCatalog().getDatabaseFile(pid.getTableId()).writePage(p);
p.markDirty(false,null);
}
}
/** Write all pages of the specified transaction to disk.
*/
public synchronized void flushPages(TransactionId tid) throws IOException {
// some code goes here
// not necessary for lab1|lab2
}
/**
* Discards a page from the buffer pool.
* Flushes the page to disk to ensure dirty pages are updated on disk.
*/
private synchronized void evictPage() throws DbException {
// some code goes here
// not necessary for lab1
PageId pid = new ArrayList<>(pageStore.keySet()).get(0);
try{
flushPage(pid);
}catch(IOException e){
e.printStackTrace();
}
discardPage(pid);
}
}
ant runsystest -Dtest=EvictionTest系统测试BUILD SUCCESSFUL。
657
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
