public class RobotDirectives extends Object
Modifier and Type | Field and Description |
---|---|
static long |
LEASE_TIME
The robots.txt lease time in milliseconds.
|
Constructor and Description |
---|
RobotDirectives(String userAgent,
long defaultDelay)
Create a new RobotDirectives instance.
|
Modifier and Type | Method and Description |
---|---|
void |
addAllowedPath(String path)
Add a path to the list of allowed paths.
|
void |
addDisallowedPath(String path)
Add a path to the list of disallowed paths
|
boolean |
allowed(String path)
Check to see if the crawler is allowed to index this path
|
void |
cleanLists()
Removes any paths in allowed that exist in disallowed
|
List<String> |
getAllowed()
Retrieve the allowed list
|
long |
getDelay()
Retrieve the delay in milliseconds
|
List<String> |
getDisallowed()
Retrieve the disallowed list
|
String |
getUserAgent()
Retrieve the user agent that this object represents
|
boolean |
isExpired()
Detect if the robots.txt lease has expired
|
void |
setDelay(float delay)
Set the delay (in seconds) to pause between requests
|
public static final long LEASE_TIME
public RobotDirectives(String userAgent, long defaultDelay)
userAgent
- the user agent string to search for in the robots.txt filepublic boolean isExpired()
public String getUserAgent()
public final void setDelay(float delay)
delay
- Delay between requestspublic void cleanLists()
public long getDelay()
public void addAllowedPath(String path)
path
- Path to addpublic void addDisallowedPath(String path)
path
- Path to addpublic List<String> getDisallowed()
public boolean allowed(String path)
path
- Path to checkCopyright © 2017. All rights reserved.